When should you use a small language model instead of a frontier model in production?

In detail

Prototype on the frontier

A frontier model — the largest, most capable general models — is the right default when you don’t yet know what your task needs. Its headroom means you can get to a working prototype without fighting the model’s limits, and it tells you what the ceiling on quality looks like before you optimize for anything else. At low volume, the per-call price is noise, so there’s no reason to trade capability for savings you won’t notice.

When a small model wins

The economics flip once a task is well-scoped and runs at high volume. A smaller model — often open-weight and fine-tuned to one job — can hit the same quality bar as a frontier model on that narrow task while costing far less per call and answering faster. The clearest version of this on the show: Intercom ran a query summarization and canonicalization step on models like GPT-4.1, a bill Fergal Reid estimates “could have been like a quarter million dollars a month,” (6:55) then replaced it with a post-trained 14-billion-parameter Qwen model that saved almost all of it. When a single task dominates your bill and its “right answer” is stable enough to tune against, that’s the signal to shrink.

Update, October 2026: Reid wrote on LinkedIn that Intercom has since switched to Nvidia and Google base models: “The important piece is a post training pipeline that works well, not any one base model.”

When the frontier still wins

Reach back for the frontier when the work is open-ended reasoning, varied, or still changing shape. A small model tuned to yesterday’s task is brittle when requirements move, and it can’t improvise the way a large general model can. Low-volume, high-stakes decisions also favor the frontier, because the cost argument for shrinking never kicks in and the capability margin is cheap insurance. This is the two-extremes reality of the current stack: broad foundations at one end, specialized efficiency at the other, and most production systems end up using both.

The migration path

The practical pattern is prototype-then-shrink: build on a frontier model, use it to define what “good enough” means with real evaluations, and only then distill or fine-tune down to a smaller model once the task is stable and the volume justifies the engineering. Do it in the wrong order — committing to a small model before you know the target — and you optimize for a bar you haven’t set yet. The frontier is how you discover the requirement; a small model is how you serve it cheaply once you have.