Skip to content

AI Glossary · Concepts

What is AI Alignment?

AI Alignment

AI alignment is the work of making an AI system pursue what its designers and users actually intend — including the goals they didn't think to spell out — rather than optimizing a literal objective in harmful or unintended ways. It spans training techniques, evaluation, and oversight.

Also known as: alignment, aligned AI

· Updated · Chain of Thought

  • 5 episodes
  • 2 sources

AI alignment concerns whether a system’s behavior matches intended goals and constraints. For a language model, distinguish the objective used during training from what happens when the deployed model generates a response. A model does not necessarily keep optimizing its training objective through new weight updates while you chat with it.

The InstructGPT paper describes one approach: supervised demonstrations followed by reinforcement learning from human feedback. The authors report improvements in human preference and truthfulness on their evaluations, while noting that the resulting models still make simple mistakes. Those results support a particular training method, not a guarantee of aligned behavior in every setting.

When the score and the goal come apart

Google DeepMind researchers define specification gaming as “a behaviour that satisfies the literal specification of an objective without achieving the intended outcome,” and note that a reward model trained on human feedback can inherit inaccuracies from that feedback (Krakovna et al., 2020). The same gap shows up in ordinary metrics. Suppose a support workflow handles ten tickets. The goal is verified resolution, with unresolved tickets escalated and kept open, but the dashboard counts tickets closed:

BehaviorClosedVerified resolved
Close all ten without solving them100
Resolve six; escalate the other four and keep them open66

The closure count ranks the first behavior higher, though it fails the goal outright. Measure verified resolution and correct escalation as their own checks, and include test cases where the easy way to raise the score breaks a constraint. Reward hacking covers the same failure during training.

For a deployed agent, make the intended behavior concrete. Specify which actions it may take, how to evaluate its responses, and when a person must review a decision. Guardrails and evaluation help test and constrain behavior; they do not by themselves solve alignment.

Sources

From the conversation