Room 4 · The Awakening · 2016
The Translator Learns with Neural Networks
The Translator Learns with Neural Networks (Google Translate team, 2016): Translating whole sentences, not loose phrases.
In 2016 Google replaced its translator's statistical method with a neural network that translates whole sentences taking their context into account. According to Google, errors fell by 55% to 85% on several language pairs, and the change reached millions of people without them having to know what had changed.
For about a decade, Google Translate worked with statistical phrase-based translation. The system learned from huge collections of translated documents, cut sentences into short phrases, translated each piece and then tried to arrange the pieces in a fluent order. It was useful, but the results often read like a patchwork, especially between languages with very different word order.
In September 2016 Google announced Google Neural Machine Translation, GNMT, and described it in the paper "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation." Instead of translating phrase by phrase, a neural network read the entire source sentence and generated the translation word by word, taking the whole context into account.
The architecture had an encoder that turned the source sentence into a sequence of vectors and a decoder that wrote the translation. Between them was an attention mechanism, first proposed for translation in 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, which let the decoder focus on the most relevant source words at each step. Both encoder and decoder used deep recurrent networks, LSTMs, and in production the system ran on Google's own specialized chips, TPUs.
In human evaluations reported in the paper, the new system reduced translation errors substantially compared with the phrase-based system on several major language pairs. It launched for Chinese to English, and within months covered many more languages. A follow-up paper that November showed a single network could translate between pairs of languages it had never seen together.
The change reached hundreds of millions of people at once, without most of them noticing. A year later, the attention mechanism would become the whole architecture: that is the first piece of Room 5.
Ask your AI: translate a sentence with an idiom into another language, then explain why a word-by-word translation would fail.
Source: arxiv.org/abs/1609.08144
See it in the museum →Cite this: Nazareno, I. (2026). The Translator Learns with Neural Networks (2016). New Mind Museum. Retrieved November 17, 2026, from https://newmindmuseum.com/museum/the-translator-learns-with-neural-networks-2016/