Skip to content

AI Glossary · Concepts

What is Small Language Model?

Small Language Model

A small language model (SLM) is a language model small enough to run cheaply, quickly or on a single device, usually tuned for a narrower set of tasks than a frontier model. There is no fixed size cutoff; the term is relative to the largest models of the day.

Also known as: SLM, SLMs, small language models

· Updated · Chain of Thought

Model ArchitectureOpen Source AI

  • 3 explainers
  • 8 episodes
  • 1 guest clip
  • 1 source

Frontier models are built to be good at nearly everything, and every answer from one costs the computing power of a very large model. A small language model trades breadth for cost and speed. With fewer parameters, it is usually cheaper and faster per answer, the smallest can run on a laptop or phone, many fit on a company’s own servers, and it is often post-trained to do one job well. Microsoft’s phi-3-mini is a well-documented case: 3.8 billion parameters, small enough to run on a phone, with reported benchmark scores in the range of Mixtral 8x7B and GPT-3.5.

The usual pattern is to prototype on a frontier model, measure quality with evals, then move stable, high-volume tasks to a small model that still passes them. Intercom did that with one summarization step on Chain of Thought’s episode 49: at 6:55, Fergal Reid describes post-training a 14-billion-parameter Qwen 3 model to summarize and canonicalize user queries before retrieval. The episode was sponsored by Galileo; the read is at 28:52. Small language model vs. frontier model covers when each wins.

Estimate the memory before you pick a device

A first estimate of the memory a model’s weights need is parameters × bytes per parameter. For phi-3-mini’s 3.8 billion parameters:

PrecisionBytes per parameterWeights
16-bit23.8 billion × 2 = 7.6 GB
4-bit0.53.8 billion × 0.5 = 1.9 GB

The Phi-3 report gives about 1.8 GB for the 4-bit model and reports it running offline on an iPhone 14 at more than 12 tokens per second. (1.9 billion bytes is about 1.8 GiB, so the figures agree.) The same arithmetic puts a 14-billion-parameter model like the one Intercom used at about 28 GB of weights at 16 bits: small next to a frontier model, but a server GPU’s job, not a phone’s.

Weights aren’t the whole bill. The cache of earlier tokens grows with context length and with the number of requests served at once, and the runtime needs its own memory. Measure the model on the target hardware with realistic prompts before committing.

Hear it from the guest

“We're going to have a compute crunch. And I think out of necessity, that's going to force us to start to work, really work on small language models.”
Watch on YouTube

Quotes lightly edited to remove filler words.

Sources

Go deeper

From the conversation