The deep learning breakthrough

Hinton and the resurrection of neural networks

In 2006, Geoffrey Hinton and his colleagues at the University of Toronto published two papers that would transform the field of machine learning. In A Fast Learning Algorithm for Deep Belief Nets (with Simon Osindero and Yee-Whye Teh) and a companion piece in Science, Hinton demonstrated that deep neural networks — networks with many hidden layers — could be effectively trained using a two-phase approach: unsupervised pre-training layer by layer, followed by supervised fine-tuning. This solved the vanishing gradient problem that had made training deep networks practically infeasible.

Why it mattered

Since Minsky and Papert's Perceptrons (1969) and the second AI winter, neural networks had been largely marginalized in favor of support vector machines and other statistical methods. Hinton's 2006 papers showed that depth — many layers of nonlinear transformations — enabled networks to learn increasingly abstract representations of data. This was the theoretical justification for what would become known as deep learning. Hinton coined the term himself and for years was largely alone in his conviction that this approach would transform AI.

Path to AlexNet and beyond

The 2006 breakthrough did not immediately dominate the field — that took until 2012, when AlexNet won the ImageNet competition by a massive margin. But it reopened research into neural networks, attracted new PhD students, and laid the groundwork for the decade of breakthroughs to come. Hinton, Yann LeCun, and Yoshua Bengio — the trio often called the "Godfathers of Deep Learning" — received the Turing Award in 2018 for their contributions.

Sources

• Hinton, G.E. et al. (2006). A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7), 1527–1554.
• LeCun, Y., Bengio, Y. & Hinton, G. (2015). Deep learning. Nature, 521, 436–444.
• Wikipedia — Deep learning