‘Attention Is All You Need’: the 2017 paper behind today’s chatbots, explained
"Attention Is All You Need", a paper by eight researchers working at Google, introduced the Transformer in June 2017, the design that sits under today's chatbots. Its central idea was to drop the step-by-step reading used by earlier language systems and let the model look at all the words of a sentence at once, deciding which ones matter for each other. That made training much faster on modern chips.
In one sentence
The Transformer is a neural network that understands each word by comparing it with every other word in the text at the same time, a process called self-attention.
How did they test it?
The authors measured the Transformer on machine translation, the standard test for language models in 2017. They used the WMT 2014 benchmarks for English-to-German and English-to-French, scored with BLEU, a measure of how closely a machine translation matches human ones. The paper was first posted on arXiv on June 12, 2017, and presented at the NIPS 2017 conference in Long Beach, California.
What did they find?
Google said the Transformer outperformed recurrent and convolutional models on both tests. It "achieves 28.4 BLEU on the WMT 2014 English-to-German translation task," and set "a new single-model state-of-the-art BLEU score of 41.8" in English to French "after training for 3.5 days on eight GPUs," according to the abstract.
What the Transformer achieved in the 2017 paper
The key was in the title. Earlier systems, called recurrent neural networks, read a sentence one word at a time, so linking two distant words took many steps. The Transformer is "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." Because every word is handled at once, the work can be split across many processors. The paper says it "allows for significantly more parallelization."
How does attention work, without the math?
Take the word "bank" in a sentence about a river. To decide what "bank" means, the Transformer "compares it to every other word in the sentence," wrote Jakob Uszkoreit, one of the authors, in a Google Research blog post on Aug. 31, 2017. The model "can learn to immediately attend to the word 'river' and make this decision in a single step." Each comparison gets a score, and the words with high scores shape the meaning.
What it doesn't say
The paper is about translation. It does not mention chatbots, and it does not predict that the design would scale into systems like ChatGPT. The results are the authors' own benchmark scores from 2017, and the blog post's claim that training is faster "by up to an order of magnitude" is also self-reported.
Context
The link to chatbots came a year later. In June 2018, OpenAI described its first GPT model by saying "first we train a transformer model on a very large amount of data in an unsupervised manner," in a post titled Improving language understanding with unsupervised learning.
The eight authors were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. A footnote says the listing order is random. All were at Google Brain or Google Research when they did the work, including Gomez, whose printed affiliation was the University of Toronto.
FAQ
What is the 'Attention Is All You Need' paper?
A 2017 paper by eight researchers working at Google that introduced the Transformer, a neural network built only on attention, the basis of today's large language models.
What is self-attention?
A method in which a model works out what each word means by comparing it with every other word in the text at once and weighting the most relevant ones.
Who wrote 'Attention Is All You Need'?
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin.
Are chatbots built on the Transformer?
OpenAI's first GPT model, described in June 2018, was a transformer trained on a large amount of unlabeled text and then fine-tuned for specific tasks.
Sources
- Vaswani et al., Attention Is All You Need (arXiv 1706.03762, NIPS 2017) · archived copy
- Google Research, Transformer: A Novel Neural Network Architecture for Language Understanding (Aug. 31, 2017) · archived copy
- OpenAI, Improving language understanding with unsupervised learning (June 11, 2018) · archived copy
Cite this
‘Attention Is All You Need’: the 2017 paper behind today’s chatbots, explained, New Mind Museum, November 6, 2026. https://newmindmuseum.com/research/2026/11/attention-is-all-you-need-transformer-paper-explained/
New Mind Museum. (2026, November 6). ‘Attention Is All You Need’: the 2017 paper behind today’s chatbots, explained. https://newmindmuseum.com/research/2026/11/attention-is-all-you-need-transformer-paper-explained/
“‘Attention Is All You Need’: the 2017 paper behind today’s chatbots, explained.” New Mind Museum, 6 Nov. 2026, https://newmindmuseum.com/research/2026/11/attention-is-all-you-need-transformer-paper-explained/.
@misc{new-mind-museum-2026-attention-is-all-you-need-the-2017-paper,
author = {{New Mind Museum}},
title = {‘Attention Is All You Need’: the 2017 paper behind today’s chatbots, explained},
year = {2026},
url = {https://newmindmuseum.com/research/2026/11/attention-is-all-you-need-transformer-paper-explained/}
}Free to quote and cite with a link to this page.