Transformer

The 2017 transformer paper combines attention with feed-forward layers and position information. Token representations are stored in tensors; attention computes weighted combinations of those representations.

For a simplified example, one attention head attending to two allowed positions might assign weights 0.8 and 0.2. Its output is 0.8 × v₁ + 0.2 × v₂, where v₁ and v₂ are their value vectors. These illustrative weights show the weighted sum; actual attention weights depend on the query and keys.

An encoder can use tokens on both sides of a position. A text-generating model such as GPT-2, a generative pre-trained transformer, uses a causal decoder, which processes the current and earlier input positions. During training, the whole sentence may already be available: in “the cat sat on the mat,” the representation at “sat” must predict “on” without reading “on the mat.” A causal mask blocks those future connections.

Full self-attention compares pairs of positions, so its computation scales quadratically with sequence length when other dimensions stay fixed. State-space models offer another approach to sequence processing. A recurrent neural network is the older step-by-step alternative, and a vision transformer applies attention to image patches instead of words.

Liquid AI’s Maxime Labonne describes a hybrid built for on-device language models in the related episode. He says it combines some attention layers with short convolutions.