BERT

BERT: understanding language in context

In October 2018, researchers at Google AI Language published BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. BERT — Bidirectional Encoder Representations from Transformers — was a new approach to pre-training language models that read text in both directions simultaneously, allowing the model to understand a word in the full context of the surrounding sentence. It set new state-of-the-art results on eleven natural language processing tasks at once.

What made BERT different

Previous language models like GPT (2018) read text left to right — each word only saw the words before it. BERT used a masked language model objective: randomly masking words in a sentence and training the model to predict them based on context from both sides. This bidirectionality was crucial for tasks requiring deep understanding, such as question answering, where the meaning of a word often depends on what comes after it. BERT also introduced next sentence prediction as a training objective, helping the model understand relationships between sentences.

Impact on search and NLP

In 2019, Google announced that BERT was being used in its search algorithm — described as "the biggest leap forward in the past five years" for search quality. BERT helped Google understand the intent behind queries rather than just matching keywords. For the NLP research community, BERT established the paradigm of pre-train then fine-tune: train a large general model on massive amounts of text, then adapt it to specific tasks with a small labeled dataset. This approach became the standard for the entire field.

Sources

• Devlin, J. et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
• Wikipedia — BERT