Compound AI Systems

A single call to a single model is rarely the whole product. A compound AI system wires together multiple pieces — several model calls, a retrieval step, tool use, validation, and the control logic that orchestrates them — into one pipeline that produces the result. RAG systems and agents are both compound by nature.

The name comes from a February 2024 Berkeley AI Research post by Matei Zaharia and colleagues, which argued that state-of-the-art results were increasingly coming from systems of interacting components rather than single models. Naming the pattern matters because it relocates where quality comes from. When the system is compound, performance depends on how the parts fit together — the retrieval, the routing, the guardrails — at least as much as on the raw model. It’s why evaluation and observability have to cover the whole pipeline, and why swapping in a better model often helps less than fixing the system around it.