“The dog bit the man” and “The man bit the dog” contain the same words. Only the order tells you who is in trouble. A transformer compares every token with every other token through attention, and that comparison on its own treats the input like an unordered bag of words. Positional encoding is the fix: each token’s embedding gets extra information that says “I am token 1”, “I am token 2”, and so on.
Approaches. The original transformer paper added fixed sine and cosine waves of different frequencies to each embedding, so every position had a distinct pattern. Some later models learned a position vector for each slot instead. Many current open models use rotary position embedding (RoPE), which rotates the vectors attention compares by an angle that depends on position, so the model sees the relative distance between two tokens.
Business example. A contract-review assistant has to know that “not” comes before “liable” in the same clause, and that the clause sits in section 4 rather than in an appendix. Order is part of the meaning, and positional encoding is how the model receives it.
Why it matters. Position handling limits how far a model’s context window can stretch. A model trained on sequences up to a certain length has never seen positions beyond it. Teams that extend context windows often change or rescale the positional scheme and then retrain on longer text. Editorial synthesis: an advertised context length is a capacity, not a guarantee that the model uses every position equally well, which is why long-context evaluation is still needed.
Sources
- Vaswani et al.: Attention Is All You Need — 2017 transformer paper, which adds sinusoidal positional encodings to token embeddings because the model contains no recurrence or convolution; checked October 24, 2026
- Su et al.: RoFormer: Enhanced Transformer with Rotary Position Embedding — Paper introducing rotary position embedding (RoPE), which encodes position by rotating query and key vectors; checked October 24, 2026