A support assistant can change behavior when its prompt changes, its model route changes or new documents enter its retrieval index. A team records those versions, runs representative evaluations before a release, and uses traces to investigate failures after deployment.
For example, a new prompt may make replies shorter while causing the assistant to omit an important returns-policy exception. A latency monitor will not detect that quality regression. An evaluation with policy exceptions can, while a trace shows which passages and tool results the model received.
MLOps supplies related practices for model development and operation. LLMOps extends the operational view to components a team may control even when it does not train the underlying model. This matters for recovery: reverting a prompt alone may not restore behavior if the retrieval index or tool contract also changed. Plan and test releases as application changes, with task outcomes as well as cost and speed visible.
Check what a partial rollback restores
Two releases of the support assistant differ in three components:
| Component | Release A | Release B |
|---|---|---|
| Prompt | P1 | P2 |
| Model configuration | M1 | M1 |
| Retrieval index | I1 | I2 |
| Tool contract | T1 | T2 |
Reverting only the prompt leaves P1 with M1, I2 and T2, which matches neither release. Find or run evaluation results for that combination before deploying it, or restore every component that changed. Restoring the old index can also be wrong if the policy documents in it are out of date; the target is correct behavior now. Reverting software also does not undo external actions the faulty release already took, such as messages it sent or records it wrote. Find and reconcile those separately.
A limited rollout can catch a bad combination before everyone gets it. Google’s SRE workbook defines canarying as “a partial and time-limited deployment of a change in a service and its evaluation,” measured against the part of the service that did not get the change (Canarying releases). For an AI application, compare task outcomes as well as errors, latency and cost, and decide in advance who can stop the expansion. The canary still serves real users, so its external actions need the same reconciliation plan.
Version labels need the same care. MLflow’s Prompt Registry keeps prompt versions immutable, but the model configuration attached to a prompt “can be changed after creation,” an alias “can point to different versions over time,” and loaded prompts are cached in memory. Log the resolved template and configuration a request actually used, beside the version number or alias.
Sources
- MLflow: Agents and LLMs — Documents tracing, evaluation, prompt management and deployment across language-model application development.
- MLflow: Prompt Registry — Prompt versions are immutable, while attached model configuration can change after creation, aliases can point to different versions, and loaded prompts are cached.
Go deeper
- MLflow: Tracing quickstart docs
Instrument an application and inspect nested model and tool calls in a trace.