Tokenization

A model reads tokens, the units a tokenizer cuts text into. Most modern tokenizers use subword methods such as byte-pair encoding, which keep common words whole and split rare ones into smaller pieces. So a common word may be a single token, while rare words, code, or other languages split into several, down to single characters. As a rough rule, a token is about three-quarters of an English word, but it varies a lot by content.

Tokens are the unit that actually matters in practice. Context windows are measured in tokens, API pricing is per token, and latency scales with how many tokens go in and come out. That’s why trimming a prompt or compacting context is a real cost and speed lever, and why text that looks short can be expensive if it tokenizes badly — dense code, unusual formatting, or non-English text all use more tokens than their length suggests.

See AI vocabulary in plain English for this term alongside the other language of models, data, agents, and infrastructure.