Standard attention compares every token with every other token, so time and memory grow with the square of the sequence length. That is the bottleneck that makes long contexts expensive in a transformer. Tri Dao and colleagues at Stanford showed in 2022 that a large part of the cost is not the arithmetic but moving data between the GPU’s large, slow high-bandwidth memory (HBM) and its small, fast on-chip SRAM (Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness). FlashAttention computes attention in tiles that fit in SRAM, so it never writes the full attention matrix out to HBM.
It is exact: the output is the same attention result, not an approximation. The paper reports a 15% end-to-end training speedup on BERT-large (sequence length 512) against the MLPerf 1.1 record, a 3x speedup on GPT-2 at sequence length 1K, and memory that grows linearly rather than quadratically with sequence length, which made longer contexts practical to train. FlashAttention-2, the following year, reorganized how work is split across the GPU and reported about a 2x speedup over the original, reaching 50 to 73% of the A100’s theoretical maximum throughput (Dao, FlashAttention-2).
You rarely call FlashAttention directly. It ships inside training frameworks and inference servers, and its effect shows up as cheaper long-context training and inference rather than as a feature. It matters for decisions you do make: how much a longer context window costs, and why kernel-level work on specific hardware can move price and speed as much as a model change. It does not change what attention does, so it is not an alternative architecture; state-space models and mixture of experts change the model itself. The model architecture topic collects the episodes on that layer, and why is it hard to move off NVIDIA? covers why hardware-specific kernels matter.
Sources
- Dao et al.: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — The 2022 paper: exact attention computed in tiles to cut traffic between GPU high-bandwidth memory and on-chip SRAM.
- Dao: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Reports about a 2x speedup over the original, reaching 50 to 73% of A100 theoretical throughput.
Go deeper
- Dao AI Lab: FlashAttention installation and features docs
Check the implementation's GPU, data-type and toolkit requirements before choosing a CUDA or ROCm backend; support differs across versions.
- PyTorch: Scaled dot product attention tutorial docs
Compare attention backends with the tutorial's timing example and inspect unsupported-kernel warnings. Timings depend on the inputs and hardware.