Explainability

Explainability is the ability to answer “why did the model do that?” in terms a human can check. For simple models you can often trace the decision to specific features. For large language models it’s much harder — billions of parameters don’t yield a clean story, and the field of interpretability is actively working on it.

The trap is mistaking a model’s self-explanation for a real one. Ask an LLM why it answered something and it will produce a fluent rationale, but that narration is generated after the fact and may not reflect the actual computation — a plausible story, not a verified cause. Turpin and colleagues showed this with chain-of-thought explanations: when they biased a prompt toward one answer, models were steered toward it and wrote step-by-step reasoning that justified it without ever mentioning the bias. Genuine explainability needs methods that probe the model’s behavior, plus, for high-stakes decisions, system design that keeps a human able to understand and override. It’s a regulatory requirement in some domains, not just a nicety: the EU AI Act gives people affected by certain high-risk AI decisions a right to a clear and meaningful explanation of the system’s role.