Attention Is All You Need
Ashish Vaswani, Noam Shazeer et al.
255
Citations
11
Influential Citations
Bioinformatics
Venue
2020
Year
Abstract Motivation An essential part of drug discovery is the accurate prediction of the binding affinity of new compound–protein pairs. Most of the standard computational methods assume that compounds or proteins of the test data are observed during the training phase. However, in real-world situations, the test and training data are sampled from different domains with different distributions. To cope with this challenge, we propose a deep learning-based approach that consists of three steps. In the first step, the training encoder network learns a novel representation of compounds and proteins. To this end, we combine convolutional layers and long-short-term memory layers so that the occurrence patterns of local substructures through a protein and a compound sequence are learned. Also, to encode the interaction strength of the protein and compound substructures, we propose a two-sided attention mechanism. In the second phase, to deal with the different distributions of the training and test domains, a feature encoder network is learned for the test domain by utilizing an adversarial domain adaptation approach. In the third phase, the learned test encoder network is applied to new compound–protein pairs to predict their binding affinity. Results To evaluate the proposed approach, we applied it to KIBA, Davis and BindingDB datasets. The results show that the proposed method learns a more reliable model for the test domain in more challenging situations. Availability and implementation https://github.com/LBBSoft/DeepCDA.
Accurate prediction of compound-protein binding affinity is critical for drug discovery, but most computational methods assume training and test data come from the same distribution. In practice, new compounds or proteins often differ from those seen during training, leading to poor generalization. DeepCDA directly addresses this real-world challenge by incorporating adversarial domain adaptation, making it highly relevant for practitioners who need robust models in dynamic drug discovery pipelines.
The paper combines multiple deep learning techniques—CNNs, LSTMs, attention, and domain adaptation—into a cohesive framework. This integration is significant because it leverages the strengths of each component: CNNs capture local substructures, LSTMs model sequential dependencies, attention highlights important interactions, and domain adaptation aligns feature distributions. The result is a method that not only predicts affinity but also adapts to new domains without requiring retraining from scratch.
The method was evaluated on three benchmark datasets: KIBA, Davis, and BindingDB. DeepCDA consistently outperformed baseline methods, including standard deep learning models without domain adaptation, especially in cross-domain settings where training and test sets had different distributions. For example, on the KIBA dataset, the model achieved higher concordance index (CI) and lower mean squared error (MSE) compared to alternatives. The authors note that the improvement is more pronounced in challenging scenarios, demonstrating the value of domain adaptation.
DeepCDA contributes to the broader AI field by demonstrating how domain adaptation can be effectively integrated into deep learning for structured biological data. This approach is not limited to drug discovery; similar techniques could be applied to other domains where distribution shifts occur, such as materials science or personalized medicine. The code is publicly available, facilitating reproducibility and further research. By addressing a practical limitation of existing methods, DeepCDA moves AI closer to reliable deployment in real-world scientific discovery.
Ashish Vaswani, Noam Shazeer et al.
Jakubův, Jan, Chvalovský, Karel et al.
Pauli Virtanen, Ralf Gommers et al.
Tom B. Brown, Benjamin Mann et al.