What is a transformer architecture?

The revolution of 2017

In 2017, Google researchers published the paper "Attention is All You Need". In it, they introduced the transformer architecture — a model that would completely overturn the AI world. Virtually every modern language model — GPT, BERT, Claude, Gemini — is built on this foundation.

What was the problem with earlier architectures?

Before the transformer, language models were built on recurrent networks (RNNs and LSTMs). These process text word by word, from left to right. This has two disadvantages: slow training (no parallelization possible) and difficulty with long-range dependencies — the relationship between word 1 and word 500 gets lost.

Illustration created with Canva AI

The core mechanism: self-attention

Self-attention lets every word in a sentence look at all other words simultaneously and determines how relevant each word is for understanding every other word. This makes it possible to capture long-range relationships — e.g. the relationship between a pronoun and the noun it refers to, even if they are far apart.

The result: the transformer can be parallelized across GPUs, which enormously accelerates training and makes scaling to models with billions of parameters possible.

Encoder, decoder, or both?

• Encoder-only (BERT) — understands text, good for classification and question answering
• Decoder-only (GPT series, Claude, LLaMA) — generates text from left to right
• Encoder-decoder (T5, BART) — combines both, good for translation and summarization

Why is the transformer so scalable?

Research showed that larger transformers on more data consistently perform better — a phenomenon called 'scaling laws'. This explains the race toward ever-larger models: GPT-3 (175 billion parameters), GPT-4 (estimated 1 trillion), and beyond.