Room 5 · Transformers · 2022
Learning from Human Preferences
Learning from Human Preferences (OpenAI · InstructGPT, 2022): Teaching a model which answers we prefer.
InstructGPT fine-tuned a language model with reinforcement learning from people's preferences: evaluators ranked answers and the model learned to produce the ones they preferred. That made models follow instructions more closely, according to the evaluators.
A model trained only to predict the next word on internet text learns to continue text, not to help. Ask it a question and it might answer, or it might write more questions, or drift into something unhelpful or harmful. In 2022 OpenAI published a widely adopted method to close that gap, in the paper "Training language models to follow instructions with human feedback," by Long Ouyang and colleagues. The resulting models were called InstructGPT.
The method had three stages. First, people wrote examples of good answers to a set of prompts, and the model was fine-tuned on them. Second, the model produced several answers to each prompt and human evaluators ranked them from best to worst. Those rankings were used to train a separate reward model that learned to predict which answers people would prefer. Third, the language model was trained with reinforcement learning to produce answers that the reward model scored highly. The technique is known as reinforcement learning from human feedback, or RLHF.
The idea had been developed in earlier work by OpenAI and DeepMind researchers, starting with a 2017 paper on learning from human preferences in games and simulated robots, and then for summarizing text.
The result surprised many people. Evaluators preferred the answers of a 1.3 billion parameter InstructGPT over those of the original 175 billion parameter GPT-3, a model more than one hundred times larger. The fine-tuned models also made up facts less often in some tests, though far from never.
RLHF has known side effects. Models can drift toward answers that please the people judging them rather than answers that are right, a tendency called sycophancy that you can read about in our Thinking with AI section.
Ask your AI: has it ever agreed with you just to be agreeable? Ask it how to tell, and how to ask so that it disagrees when it should.
Source: arxiv.org/abs/2203.02155
See it in the museum →← AlphaFold Solves Protein Folding · Images from Text →
Cite this: Nazareno, I. (2026). Learning from Human Preferences (2022). New Mind Museum. Retrieved November 17, 2026, from https://newmindmuseum.com/museum/learning-from-human-preferences-2022/