Frontier models are built to be good at nearly everything, and every answer from one costs the computing power of a very large model. A small language model trades breadth for cost and speed. With fewer parameters, it is usually cheaper and faster per answer, the smallest can run on a laptop or phone, many fit on a company’s own servers, and it is often post-trained to do one job well. Microsoft’s phi-3-mini is a well-documented case: 3.8 billion parameters, small enough to run on a phone, with reported benchmark scores in the range of Mixtral 8x7B and GPT-3.5.
The usual pattern is to prototype on a frontier model, measure quality with evals, then move stable, high-volume tasks to a small model that still passes them. Intercom did that with one summarization step on Chain of Thought’s episode 49: at 6:55, Fergal Reid describes post-training a 14-billion-parameter Qwen 3 model to summarize and canonicalize user queries before retrieval. The episode was sponsored by Galileo; the read is at 28:52. Small language model vs. frontier model covers when each wins.
Estimate the memory before you pick a device
A first estimate of the memory a model’s weights need is parameters × bytes per parameter. For phi-3-mini’s 3.8 billion parameters:
| Precision | Bytes per parameter | Weights |
|---|---|---|
| 16-bit | 2 | 3.8 billion × 2 = 7.6 GB |
| 4-bit | 0.5 | 3.8 billion × 0.5 = 1.9 GB |
The Phi-3 report gives about 1.8 GB for the 4-bit model and reports it running offline on an iPhone 14 at more than 12 tokens per second. (1.9 billion bytes is about 1.8 GiB, so the figures agree.) The same arithmetic puts a 14-billion-parameter model like the one Intercom used at about 28 GB of weights at 16 bits: small next to a frontier model, but a server GPU’s job, not a phone’s.
Weights aren’t the whole bill. The cache of earlier tokens grows with context length and with the number of requests served at once, and the runtime needs its own memory. Measure the model on the target hardware with realistic prompts before committing.
Hear it from the guest
“We're going to have a compute crunch. And I think out of necessity, that's going to force us to start to work, really work on small language models.”
Quotes lightly edited to remove filler words.
Sources
- Abdin et al. (Microsoft): Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone — Introduces phi-3-mini, a 3.8-billion-parameter model trained on 3.3 trillion tokens that rivals Mixtral 8x7B and GPT-3.5 on reported benchmarks while being small enough to run on a phone. Quantized to 4 bits it occupies about 1.8 GB and ran fully offline on an iPhone 14 at more than 12 tokens per second; checked November 7, 2026.
Go deeper
- When should you use a small language model instead of a frontier model in production? AI, decoded · Small Language Model vs. Frontier Model
- Are small language models better than large ones for production? AI, decoded · Small Language Models in Production
- Can switching to an open model actually cut your AI costs? AI, decoded · When an Open Model Beats a Frontier API on Cost
From the conversation
-
How Intercom Cut $250K/Month by Ditching GPT for Open Models | Fergal Reid -
I Built an AI Coworker That Runs 90% of My Day | Sterling Chin, Postman -
The AI Agent Trust Gap: Bridging Risk to Reliability | Elastic’s Philipp Krenn -
Explaining Eval Engineering | Galileo's Vikram Chatterji -
Architecting Reliable Agentic AI | Cisco’s Giovanna Carofiglio on the AGNTCY Collective -
From Demo to Defensibility: How to Build an AI Business that Lasts | Aurimas Griciūnas -
The Emerging AI Agent Stack | CrewAI’s João Moura -
Vercel's Playbook for AI Agents: From Vibe Check to Production | Malte Ubl