Conference Paper
Natural Language Processing
Featured

Attention Is All You Need

Ashish Vaswani(Google Brain), Noam Shazeer(Google Brain), Niki Parmar(Google Research), Jakob Uszkoreit(Google Research), Llion Jones(Google Research), Aidan N. Gomez(University of Toronto), Lukasz Kaiser(Google Brain), Illia Polosukhin(Google Research)
June 12, 2017NeurIPS 2017130,000 citations
arXivPDFDOICode

130k

Citations

12k

Influential Citations

NeurIPS 2017

Venue

2017

Year

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature.

Analysis

Why This Paper Matters

"Attention Is All You Need" introduced the Transformer architecture, which has become the foundation for virtually all modern large language models, including GPT, BERT, T5, PaLM, LLaMA, and Claude. Before this paper, the dominant approach to sequence modeling relied on recurrent neural networks (RNNs) and LSTMs, which processed tokens sequentially — making them slow to train and limited in their ability to capture long-range dependencies.

Technical Contributions

The paper's central insight is that self-attention alone can replace recurrence for sequence transduction tasks. The key innovations include:

  • Multi-head attention: Instead of a single attention function, the model uses multiple parallel attention heads, each learning different types of relationships between tokens.
  • Positional encodings: Since the architecture has no recurrence, sinusoidal positional encodings inject information about token positions.
  • Scaled dot-product attention: The attention mechanism uses scaling by the square root of the key dimension to prevent softmax saturation.

Results

The Transformer achieved state-of-the-art results on WMT 2014 translation benchmarks:

  • 28.4 BLEU on English-to-German (improving over previous best by 2+ BLEU)
  • 41.8 BLEU on English-to-French (new single-model SOTA)
  • Training time was a fraction of competing models

Legacy

This paper has been cited over 130,000 times and fundamentally reshaped the field. The Transformer architecture is now the backbone of generative AI, powering applications from chatbots and code generation to image synthesis and protein structure prediction.

transformerself-attentionencoder-decodermachine-translationneural-networksdeep-learning
TransformersAttention MechanismsNatural Language ProcessingMachine Translation

Related Papers