Take the sentence “The trophy didn’t fit in the suitcase because it was too big.” To work out what “it” refers to, a reader looks back at “trophy” and “suitcase” and decides which one fits. Attention gives a model a way to do something similar: for every word, it scores every other word for relevance and blends in information from the highest-scoring ones.
Self-attention is one kind of attention: it relates positions within the same sequence. Cross-attention instead connects one sequence with another, such as a translation decoder attending to the source sentence. These are different uses of the broader mechanism.
Attention became influential in machine translation through Bahdanau and colleagues’ 2014 work. The 2017 paper “Attention Is All You Need” built the transformer architecture around it, and transformers now underpin most large language models.
Why it matters
Attention is a big reason modern models handle context so well. It is also expensive: in a standard transformer the work grows roughly with the square of the input length, which is why long context windows cost more and why engineers use tricks such as the KV cache.
On the podcast
In episode 43, Liquid AI’s Maxime Labonne explains that the company’s small on-device models use a hybrid design that keeps only a few attention layers at 2:38. He adds that when researchers go back to basic mathematical operations, “the attention layer is super strong, actually,” a strong baseline to beat at 6:11. Those are his engineering observations.
Related terms
The transformer is the architecture built around attention. Neural machine translation is where attention was first popularized.
Hear it from the guest
“The architecture that we ended up with is a hybrid that has some attention layers.”
“The attention layer is super strong, actually.”
Quotes lightly edited to remove filler words.
Sources
- Bahdanau, Cho and Bengio: Neural Machine Translation by Jointly Learning to Align and Translate — Introduces a mechanism that lets a translation model softly search for the relevant parts of the source sentence for each output word; checked October 24, 2026.
- Vaswani et al.: Attention Is All You Need — Proposes the Transformer, an architecture based solely on attention mechanisms, and describes scaled dot-product and multi-head attention; checked October 24, 2026.
