Room 5 · Transformers · 2020
GPT-3 and the Scaling Laws
GPT-3 and the Scaling Laws (OpenAI, 2020): Bigger, and suddenly more capable.
GPT-3 had 175 billion parameters, more than a hundred times those of GPT-2 (1.5 billion). It showed that a large model could solve new tasks from a few examples written in the same text, without retraining. The “scaling laws” were also published that year: performance improves predictably as the model, the data and the compute grow.
In February 2019 OpenAI presented GPT-2, a model with 1.5 billion parameters, and released it in stages over that year, citing concerns about misuse. In May 2020 it described GPT-3, with 175 billion parameters, in a paper titled "Language Models Are Few-Shot Learners."
The title states the main finding. Earlier models had to be fine-tuned with labeled data for each new task. GPT-3 could often perform a task just from a description and a few examples written in the prompt, with no change to its weights. Show it a few English sentences followed by their French translations, and it would continue the pattern with a new sentence. The same model wrote short articles, answered questions, solved simple arithmetic and wrote code, with uneven quality but surprising breadth. Researchers call this in-context learning.
In January 2020, a few months earlier, a group at OpenAI led by Jared Kaplan had published "Scaling Laws for Neural Language Models." It showed that a model's error falls smoothly and predictably as three things grow: the number of parameters, the amount of training data and the computing power used. The relationships were steady across many orders of magnitude. That turned scale into something you could plan and invest in, and GPT-3 was its most visible demonstration.
In 2022 a DeepMind study known as Chinchilla refined the recipe: for a given compute budget, many models were too large and trained on too little data.
OpenAI did not release GPT-3's weights. Access came through a paid API launched in June 2020, which allowed thousands of developers to build products on top of a model they never saw. That model of access became common in the industry.
Ask your AI: give it two examples of a pattern you invent and see whether it continues correctly. That is in-context learning.
Source: arxiv.org/abs/2005.14165
See it in the museum →← BERT and GPT: Pretrain and Fine-tune · AlphaFold Solves Protein Folding →
Cite this: Nazareno, I. (2026). GPT-3 and the Scaling Laws (2020). New Mind Museum. Retrieved November 17, 2026, from https://newmindmuseum.com/museum/gpt-3-and-the-scaling-laws-2020/