Room 5 · Transformers · 2017
Attention Is All You Need
Attention Is All You Need (Vaswani and seven other authors · Google, 2017): The architecture behind today's language models.
A Google team presented the Transformer, an architecture that replaces word-by-word reading with an attention mechanism: each word is compared with all the others in the sentence and weighted by how much each one matters. It trains in parallel and scales much better than the earlier recurrent networks.
In June 2017 eight researchers, most of them at Google Brain and Google Research, posted a paper with an unusually confident title: "Attention Is All You Need." The authors were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin. The paper listed them as equal contributors, in random order.
At the time, the best translation systems, like the one at the end of Room 4, read sentences one word at a time with recurrent networks and used attention as a helper. The Transformer removed the recurrence entirely and kept only attention. In self-attention, every word in a sentence computes how relevant every other word is to it, and builds its new representation as a weighted mix of all of them. In the sentence "the trophy didn't fit in the suitcase because it was too big," attention is the mechanism that lets "it" connect strongly with "trophy." The model runs several of these attention patterns in parallel, called heads, and stacks many layers of them.
Because all the words are processed at the same time rather than in sequence, Transformers make full use of GPUs and can be trained on far more data. In the paper, the model set new records in English to German and English to French translation while needing a fraction of the training cost of earlier systems.
The Transformer quickly spread beyond translation: to text, then images, audio, proteins and code. The "T" in GPT stands for Transformer, and nearly every large language model today is built on this design.
Over the following years, most of the eight authors left Google to found or join AI companies.
Ask your AI: in the sentence about the trophy and the suitcase, what changes if "big" becomes "small"? Ask it to explain using attention.
Source: arxiv.org/abs/1706.03762
See it in the museum →BERT and GPT: Pretrain and Fine-tune →
Cite this: Nazareno, I. (2026). Attention Is All You Need (2017). New Mind Museum. Retrieved November 17, 2026, from https://newmindmuseum.com/museum/attention-is-all-you-need-2017/