An alert needs a named signal and a baseline. A change in incoming ticket categories, a change in predicted labels and an increase in reviewed errors answer different questions. Some distribution shifts are harmless. Rabanser et al.’s shift-detection study treats detecting a shift and assessing its harm as separate tasks.
Separate the change from its measured effect
- Input distribution: Are you receiving different kinds of requests from the reference window? An alert can identify a change before outcome labels arrive.
- Prediction distribution: Is the model producing a different mix of answers? That can reflect a different workload, a changed system or a quality problem; investigate the examples.
- Prediction quality: How often are predictions correct on current cases? For classifier accuracy, compare predictions with trusted outcome labels. AWS’s model-quality documentation gives an implementation example of that join.
Data drift concerns the input distribution. Concept drift often refers to a changing relationship between inputs and target outcomes, as described in Gama et al.’s survey. Terminology varies, so state the quantity your monitor compares. A distribution alert alone does not establish a drop in accuracy, and a quiet alert does not certify quality.
Check the traffic mix
Consider an invented ticket classifier with two request types. Its accuracy is held at 90% for billing and 50% for bug reports in both windows. Only the number of tickets of each type changes. These are assigned counts, not a measured model result.
| Ticket type | Baseline | Later |
|---|---|---|
| Billing | 72 / 80 | 18 / 20 |
| Bug reports | 10 / 20 | 40 / 80 |
| All tickets | 82 / 100 | 58 / 100 |
Overall accuracy falls from 82% to 58% because the workload contains more of the harder ticket type. Both within-type rates stay fixed in this example. Looking only at the overall score would hide that explanation.
Does the lower overall accuracy prove concept drift?
No. The example changes only traffic shares while holding the two within-type accuracy rates fixed. Aggregate accuracy alone cannot identify which mechanism changed. In real traffic, inspect the affected cases and outcome evidence rather than assuming that unchanged averages within broad groups prove an unchanged input-target relationship.
Choose the next evidence
- Record the reference window and system versions, including the model, prompts, preprocessing and retrieval where relevant. Check for a release or data-pipeline change before attributing the alert to the outside world.
- Inspect affected requests and compare meaningful slices such as request type or language. A change in one monitored feature does not describe every possible shift.
- Join predictions to representative, reviewed outcomes when available. Track quality by slice as well as overall. Unlabeled monitors can estimate risk under assumptions; keep those estimates distinct from directly measured accuracy.
- Choose a response from the evidence. A broken input pipeline may need a data fix; changed task requirements may need new labels and evaluation. Evaluate a candidate change before rollout instead of treating every alert as an instruction to retrain.
Observability helps locate what changed; ongoing evaluation checks its effect on the task. A passing pre-launch test is a baseline to revisit as production changes.
Sources
- AWS: Model quality monitoring — An implementation example of joining captured predictions with actual labels to measure quality. This is an explanatory reference, not a recommendation to adopt the service.
- Gama et al.: A Survey on Concept Drift Adaptation — Author-uploaded survey copy. Its abstract and problem-setting discussion describe changes in the input-target relationship and distinguish different forms of drift.
Go deeper
- What is AI observability, and why do you need it in production? AI, decoded · What Is AI Observability
- How do you evaluate an AI agent? AI, decoded · How to Evaluate an AI Agent: Steps, Outcomes and Side Effects
- Rabanser et al.: Failing Loudly paper
Compare the dataset-shift detectors and the study's assessment of whether shifts harm predictions. Inspect the experimental datasets and perturbations before choosing a monitoring approach.