Room 3 · The Winters · 1986

Backpropagation

Backpropagation (Rumelhart, Hinton and Williams, 1986): How to train multi-layer networks.

In 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams published in Nature a method to train networks with hidden layers: the error is measured at the output and distributed backwards, layer by layer, to correct the weights. It solved the problem Minsky and Papert had pointed out in 1969.

On October 9, 1986, the journal Nature published a four-page letter by David Rumelhart, Geoffrey Hinton and Ronald Williams: "Learning representations by back-propagating errors." It answered the question left open by the book Perceptrons in Room 2: how do you train a network with hidden layers?

The method works in two passes. First, an example goes forward through the network and produces an output. The output is compared with the right answer, and the difference is measured as an error. Then the error is sent backward, layer by layer, using the chain rule of calculus to work out how much each connection contributed to it. Every weight is nudged a little in the direction that reduces the error. Repeat this over many examples and the network gradually improves.

The paper's title stresses an important result: the hidden layers learned useful internal representations on their own. Nobody told the network which features to look for. It discovered them because they helped reduce the error. Those learned representations are what makes deep learning powerful.

The mathematics had a history. Seppo Linnainmaa described the general method for computing derivatives backward in 1970, and Paul Werbos proposed applying it to neural networks in his 1974 doctoral thesis. What the 1986 paper did was demonstrate clearly that it worked and make it known to a wide audience, together with the Parallel Distributed Processing books published the same year.

Computers of the time were too slow and data too scarce to go very far. But every large model trained today, including the language models you talk to, still learns by backpropagation.

Ask your AI: explain backpropagation with a network of only two weights and real numbers. No jargon allowed.

Did you know? The idea had precursors: Paul Werbos had proposed it in his 1974 thesis.

Source: doi.org/10.1038/323533a0

See it in the museum →

← Hopfield Networks · The Second Winter →

All of Room 3: The Winters

Cite this: Nazareno, I. (2026). Backpropagation (1986). New Mind Museum. Retrieved November 17, 2026, from https://newmindmuseum.com/museum/backpropagation-1986/