Training-Serving Skew

A food-delivery company trains a model to predict delivery times. In training, the “restaurant busyness” feature came from a nightly batch job that counted orders per hour. In production, the app computes it live, counting only the last 15 minutes. The numbers mean different things, so a model that scored well offline makes poor predictions for real customers. Nothing is wrong with the model; the two pipelines disagree.

Google’s Rules of Machine Learning lists three causes: different processing code in the training and serving pipelines, data that changed between training and serving, and feedback loops where the model’s own decisions change the data it later sees. The first is the easiest to prevent. Share one feature-computation path between training and serving, log the features actually used at serving time, and train future models on those logs.

Skew is a mismatch between training and serving inputs or processing; model drift describes changes in the data or relationships a deployed model encounters over time, and the two can overlap. Both show up as a gap between offline evaluation and live results, and both are caught by monitoring input distributions against the training data, which tools such as TensorFlow Data Validation automate.

Editorial note: language-model applications have a version of the same problem, though the term comes from classic ML. A model fine-tuned with one prompt template or chat format and then served with another can behave worse than its tests suggested. Keeping templates in prompt management and rerunning regression tests in the serving setup catches it. See MLOps for the surrounding practice.