A team evaluating a RAG assistant might compare retrieval plus reranking with retrieval alone. Keeping the generator, questions and scoring rules fixed helps reveal whether the reranker improves answers enough to justify its latency. This is an illustrative experiment, not a reported result.
Removing a component measures its contribution within that particular system. It does not show that the component is useless in every design: other parts may compensate for it, or depend on it. Changing the dataset and several components at once makes attribution harder.
State whether each variant is retrained or retuned. Disabling part of a fixed model and training a replacement without it answer different questions. Report the evaluation conditions and variation across runs so a small apparent gain does not become an unsupported architectural claim.
Remove components one at a time and together
Extend the retrieval example to two optional components, query rewriting and reranking. Each variant answers the same 100 questions with the same pass criterion:
| Query rewriting | Reranking | Passing answers |
|---|---|---|
| On | On | 90 |
| Off | On | 89 |
| On | Off | 88 |
| Off | Off | 70 |
Removing rewriting alone costs one point and removing reranking alone costs two, yet removing both costs 20. Each component covers for the other here, so the single removals understate what the pair contributes, and dropping both because each looked cheap would be a mistake. Run the pair as well as each single removal, compare the questions each variant fails as well as the totals, and repeat runs when outputs vary before reading meaning into a one-point gap.
Sources
- Meyes et al.: Ablation Studies in Artificial Neural Networks — Sections 3.1–3.3 disable units, evaluate on shared test data and study recovery training separately.
Go deeper
- Meyes et al.: Ablation study code docs
Inspect the authors’ code and saved networks for reproducing component-removal experiments.