Are small language models better than large ones for production?

In detail

Big models are generalists; production tasks are specific

A frontier model is built to do almost anything reasonably well. Most production features don’t need that — they do one task, over and over. For a narrow, repeated job, a smaller model tuned on that task can hit the same quality as a giant general one, because it only has to be good at the thing you actually run.

What small buys you

Three things the big general model can’t match. Cost: a smaller model can be far cheaper per call, which is why Intercom’s switch of one high-volume step from GPT-4.1 to a tuned smaller model cut spend so sharply. Fergal Reid frames the goal at 43:55: if 30% or 50% of Intercom’s tokens can run on its own lighter model, that is real margin and lower latency. Latency: fewer parameters usually means faster responses, which matters for anything interactive. Control: small enough models can run on your own infrastructure, keeping data in-house and removing a per-token dependency on an outside provider.

When to stay with a large model

Small isn’t always the answer. Open-ended reasoning, tasks with huge or unpredictable input variety, and work where you can’t gather enough examples to tune still favor a frontier model. Knowledge is the hard limit: Liquid AI’s Maxime Labonne says in episode 43 at 36:43 that knowledge depends on parameter count, and distilling into a one-billion-parameter model will never make it as smart as a three-billion one. Small models can be strong on benchmarks. Microsoft’s phi-3-mini, at 3.8 billion parameters and small enough to run on a phone, reported benchmark scores in the range of Mixtral 8x7B, which has about 47 billion parameters, and GPT-3.5. New architectures, like the efficiency-focused designs Liquid AI is exploring, are widening what small models can do — but the rule holds: match the model to the task. Databricks’ Denny Lee puts it as “there isn’t one Uber model for your company. You’re building many smaller models” (episode 22 at 20:47). NVIDIA researchers make the agentic version of the case in a 2025 position paper, arguing that agents mostly run a small number of specialized, repetitive tasks, where small models are sufficient and more economical, with larger models invoked selectively where general reasoning or open-domain conversation is essential.

The decision rule

Don’t default to the biggest model. For each task, find the smallest model that still passes your evals. That’s usually where cost, speed, and quality balance out in production. Small language model vs. frontier model goes deeper on the comparison, and can an open model cut your AI costs? covers hosting one yourself.

Disclosure: Episode 22 aired while Chain of Thought was Galileo’s podcast and Conor worked there (42:47); Databricks, the guest’s employer, is an investor in and partner of Galileo (13:44). Episode 49 was sponsored by Galileo, the Season 3 presenting sponsor (read at 28:52).