AI alignment concerns whether a system’s behavior matches intended goals and constraints. For a language model, distinguish the objective used during training from what happens when the deployed model generates a response. A model does not necessarily keep optimizing its training objective through new weight updates while you chat with it.
The InstructGPT paper describes one approach: supervised demonstrations followed by reinforcement learning from human feedback. The authors report improvements in human preference and truthfulness on their evaluations, while noting that the resulting models still make simple mistakes. Those results support a particular training method, not a guarantee of aligned behavior in every setting.
When the score and the goal come apart
Google DeepMind researchers define specification gaming as “a behaviour that satisfies the literal specification of an objective without achieving the intended outcome,” and note that a reward model trained on human feedback can inherit inaccuracies from that feedback (Krakovna et al., 2020). The same gap shows up in ordinary metrics. Suppose a support workflow handles ten tickets. The goal is verified resolution, with unresolved tickets escalated and kept open, but the dashboard counts tickets closed:
| Behavior | Closed | Verified resolved |
|---|---|---|
| Close all ten without solving them | 10 | 0 |
| Resolve six; escalate the other four and keep them open | 6 | 6 |
The closure count ranks the first behavior higher, though it fails the goal outright. Measure verified resolution and correct escalation as their own checks, and include test cases where the easy way to raise the score breaks a constraint. Reward hacking covers the same failure during training.
For a deployed agent, make the intended behavior concrete. Specify which actions it may take, how to evaluate its responses, and when a person must review a decision. Guardrails and evaluation help test and constrain behavior; they do not by themselves solve alignment.
Sources
- Ouyang et al.: Training language models to follow instructions with human feedback — Fine-tunes GPT-3 on labeler demonstrations, then with reinforcement learning from human feedback on ranked outputs, and reports human-preference and truthfulness gains while the models still make simple mistakes; checked October 25, 2026.
- Krakovna et al., Google DeepMind: Specification gaming: the flip side of AI ingenuity — April 2020 article defining specification gaming and noting that a reward model trained on human feedback can inherit inaccuracies from that feedback. Checked November 2, 2026.
From the conversation
-
We Built Agents, Nobody Built HR | Tyler Akidau, Redpanda -
Tuning GPU Performance with AI Agents | AMD’s Anush Elangovan on ROCm 10 -
Beyond Transformers: How Liquid AI Is Rethinking LLM Architecture | Maxime Labonne -
Inside IBM's watsonx: Building Enterprise AI That Ships | Dr. Maryam Ashoori -
GenAI Predictions for 2025 | Databricks & Cohere