A travel site wants to know whether a new model writes better hotel summaries. Offline evals look good, but the real question is whether people book more. The team sends a random 10% of visitors to pages with the new summaries, keeps everyone else on the old ones, and after two weeks compares booking rates. If B is better by more than chance would explain, it ships; if not, it doesn’t, however good the demo looked.
The rules that make a test trustworthy apply to AI too. Choose the metric and the sample size before starting. Randomize by user, not by request, so one person doesn’t see both versions. Watch guardrail metrics, such as complaints, latency or cost, alongside the main one, and don’t stop early the moment the numbers look good. AI changes add a wrinkle: a model can raise one metric by gaming it, for example deflecting support tickets instead of resolving them.
In episode 49 at 22:07, Intercom’s Fergal Reid says the company A/B tests every model before production. It checks that soft resolutions rise while hard resolutions don’t fall, to avoid building “a deflection machine”, and its switch between model providers was decided by a test across millions of end-user interactions. He calls production testing with real users the gold standard because backtests and evals can diverge from production, and he reports a surprise: higher latency tended to produce more hard resolutions. Those are Intercom’s findings at its scale. The episode was sponsored by Galileo; the read is at 28:52.
Most teams don’t have that traffic. Smaller ones lean more on AI evaluation before release, then use a canary deployment or shadow deployment to limit risk while collecting production evidence.
Hear it from the guest
“And then we always AB test any model before putting it in production.”
Quotes lightly edited to remove filler words.
Sources
- Tang et al.: Overlapping Experiment Infrastructure, More, Better, Faster Experimentation (Google, KDD 2010) — Google evaluates almost every user-affecting change by experiment, including machine learning changes to ranking; the paper describes infrastructure for running many experiments at once; checked October 24, 2026.
- Microsoft Research: Patterns of Trustworthy Experimentation, Pre-Experiment Stage — Microsoft's Experimentation Platform team on how data-quality issues and design or interpretation biases lead A/B tests to incorrect conclusions; checked October 24, 2026.
