Early chat models had context windows of a few thousand tokens, roughly a few pages. Long-context models raised that by orders of magnitude, so instead of chunking a 300-page contract for retrieval-augmented generation, you can paste the whole thing and ask questions. That simplifies some pipelines and makes others possible, such as reasoning across an entire repository.
It isn’t free. Attention cost grows with input length, the KV cache grows with every token held, and per-token pricing means a full window is expensive to send on every turn. Recall also varies: the “lost in the middle” research found models often miss facts buried mid-input, and benchmark wins on synthetic needle-in-a-haystack tests don’t always carry over to messy real documents. Many teams combine long context with retrieval and context engineering rather than choosing one.
In episode 12 at 21:22, Google’s Logan Kilpatrick gives long context as an example of a capability the Gemini team knew developers needed, then worked on both the algorithms and the infrastructure to run it at scale. At 25:46 he credits TPUs as the reason it could reach production at that scale. The episode was sponsored by Galileo (mention at 18:07).
Architecture choices change the cost. In episode 43 at 3:08, Liquid AI’s Maxime Labonne says replacing many attention layers with short convolution layers lowers memory use with long context, so the KV cache doesn’t explode.
Hear it from the guest
“Let's go figure out how to make the algorithmic progress in order to enable long context and then, like, the infrastructure work in order to make that actually work at scale in production.”
Quotes lightly edited to remove filler words.
Sources
- Gemini Team: Gemini 1.5, Unlocking multimodal understanding across millions of tokens of context (arXiv 2403.05530) — Google DeepMind's technical report on Gemini 1.5, reporting near-perfect retrieval (above 99%) up to at least 10 million tokens in research testing; checked October 24, 2026.
- Liu et al.: Lost in the Middle, How Language Models Use Long Contexts (arXiv 2307.03172) — Finds that performance often drops when relevant information sits in the middle of a long input rather than at the start or end; checked October 24, 2026.
