Room 5 · Transformers · 2018
BERT and GPT: Pretrain and Fine-tune
BERT and GPT: Pretrain and Fine-tune (Google and OpenAI, 2018): First train on huge amounts of text, then adjust to a task.
In 2018 OpenAI presented GPT and Google presented BERT. Both are first trained on enormous amounts of text, and then adjusted to specific tasks. BERT learns by filling in masked words; GPT, by predicting the next word.
In 2018 two papers turned the Transformer into a general recipe for language. Both followed the same plan: first pretrain a large model on a mountain of ordinary text, with no human labels, and then fine-tune it on a specific task with a small amount of labeled data.
In June 2018 Alec Radford and colleagues at OpenAI published "Improving Language Understanding by Generative Pre-Training," the model now known as GPT. It was trained on thousands of unpublished books to predict the next word, reading only from left to right. In October 2018 Jacob Devlin and colleagues at Google published BERT, short for Bidirectional Encoder Representations from Transformers. BERT read text in both directions and learned by filling in words that had been hidden, a little like a cloze exercise in a language class.
The effect was immediate. Fine-tuned versions of BERT set new records on a wide set of language benchmarks, including question answering and inference tests. Within months, researchers everywhere were downloading the released models and adapting them to their own problems, from legal documents to medical records. Building a new language system from scratch for every task suddenly looked wasteful.
The two approaches later diverged. Models like BERT, which read in both directions, became the workhorses of search, classification and analysis. Models like GPT, which predict the next word, turned out to scale into systems that can write, converse and follow instructions. That second branch leads to the rest of this room.
In October 2019 Google said it was using BERT to understand about one in ten English searches in the United States.
The names are no accident: BERT followed ELMo, an earlier model from the Allen Institute, both characters from Sesame Street.
Ask your AI: what is the difference between filling in a hidden word and predicting the next one? Which one is it doing right now?
Source: arxiv.org/abs/1810.04805
See it in the museum →← Attention Is All You Need · GPT-3 and the Scaling Laws →
Cite this: Nazareno, I. (2026). BERT and GPT: Pretrain and Fine-tune (2018). New Mind Museum. Retrieved November 17, 2026, from https://newmindmuseum.com/museum/bert-and-gpt-pretrain-and-fine-tune-2018/