Almost every model in the current wave — language, image, audio — is a transformer, and the transformer's one big idea has a plain-English name: attention. When the model processes a word, attention lets it look at every other word in the input and decide which ones matter for understanding this one, right now. That selective focus is the mechanism the whole modern stack is built on.
The pronoun test
The classic example is pronoun resolution. In "the trophy didn't fit in the suitcase because it was too big," what does "it" refer to? Answering that means weighing "trophy" against "suitcase" using the rest of the sentence. Attention is exactly the mechanism that lets each word pull in context from the words relevant to it — however far away they sit in the sequence.
Why it won: parallelism
The reason this beat what came before is almost boring, but it was decisive: parallelism. Older recurrent models read a sequence one step at a time, which is inherently serial and slow to train. A transformer looks at the whole sequence at once, which maps beautifully onto GPUs — and that trainability, more than any single accuracy win, is why the architecture took over so completely.
The quadratic catch
It isn't free. Attention compares every position with every other position, so the cost grows with the square of the sequence length. That quadratic scaling is precisely why context windows were historically limited and why so much research targets making long-context attention cheaper. Next time you read about efficiency work on long inputs, this is usually the wall being pushed on.
You don't need the matrix math
You don't need the linear algebra to reason about these systems well. Holding the intuition — the model learns what to pay attention to, and it can attend to everything at once — is enough to make good engineering decisions, and it turns a stack that can feel like magic into something you can actually think clearly about.
Sources & further reading
- Vaswani et al., Attention Is All You Need (2017) — the paper that introduced the transformer.
- Alammar, The Illustrated Transformer — the clearest visual walkthrough of the mechanism.
- Tay et al., Efficient Transformers: A Survey (2020) — the research pushing on that quadratic wall.
Editorial note — An educational explainer on the transformer architecture introduced by Vaswani et al. (2017). Simplified by design — it deliberately omits the underlying matrix math and quotes no benchmark figures.
