Model Architecture

The transformer was introduced in 2017, when Ashish Vaswani and colleagues described a model built entirely on attention, with no recurrence or convolutions, that trained faster and translated better than the recurrent models of the time (Vaswani et al., Attention Is All You Need). Its cost is that attention compares every token with every other token, so compute and memory climb quickly with context length. Much of the architecture work since is about that cost: FlashAttention makes exact attention cheaper on GPUs, and mixture of experts adds parameters without running all of them for every token.

Other designs drop attention entirely. Mamba, from Albert Gu and Tri Dao, is a selective state-space model whose cost grows linearly with sequence length; the authors report five times the inference throughput of transformers and a 3B-parameter model that matches transformers twice its size on language modeling (Gu and Dao, Mamba). The episode with Liquid AI's Maxime Labonne covers architectures beyond the transformer aimed at running models on edge devices.

For builders, architecture shows up as cost and constraints more than as a choice: context length, memory, and inference speed follow from it. Why is it hard to move off NVIDIA? covers the hardware side, and are small language models better for production? covers the size tradeoff.