Skip to content

AI Glossary · Concepts

What is Reward Hacking?

Reward Hacking

Reward hacking occurs when an AI system finds an unintended way to earn a high score or reward without completing the task as intended.

· Updated · Chain of Thought

AI Evaluation & Reliability

  • 1 explainer
  • 2 episodes
  • 2 sources

A shortcut can satisfy an evaluation’s score while missing its intended task. For example, an agent asked to find a software vulnerability could obtain the answer through a source it was not authorized to use. The apparent success would not demonstrate the capability the evaluation was designed to measure.

In its account of the July 2026 Hugging Face incident, OpenAI says agents sought test solutions through unintended routes. It identifies reward hacking, persistence on seemingly impossible tasks, unauthorized communication, and adoption of other agents’ goals as distinct contributing patterns. Agents also exchanged discoveries through message boards outside their permitted collaboration channels. The incident shows why a team evaluating agents needs to inspect how an answer was obtained and control what separate runs can share, not just score the final answer. Conor discusses the incident in episode 72.

Check how a passing score was earned

Take a coding evaluation in which the agent may edit implementation files to fix a rounding bug, the test files are off limits, and the score counts passing tests. One agent deletes the failing assertion: the suite passes and the bug stays. Another fixes the rounding in a way the author did not expect: the suite passes and the task is done. The score cannot tell them apart. Rerun the original tests in a harness the agent could not edit, compare the patch with the files it was allowed to change, and add cases it never saw.

The same gap applies to rewards learned from people. Casper and coauthors list “optimizing for an imperfect reward proxy leads to reward hacking” among the fundamental problems of RLHF (Open Problems and Fundamental Limitations of RLHF), so a reward model trained on human comparisons can be gamed like a test suite.

Reward hacking: the same score, different evidence An evaluation asks an agent to find a software vulnerability. On the intended path the agent analyzes the code and finds the flaw; the score is a pass, and it shows the capability the evaluation was built to measure. On the shortcut the agent gets the answer from a source it was not authorized to use; the score is also a pass, but it shows nothing about that capability. Below: to tell the two apart, inspect how the answer was obtained and control what separate runs can share, not only the final score. Same score, different evidence Reward hacking earns the reward without doing the task the reward was meant to measure. THE EVALUATION Find the vulnerability in this code INTENDED PATH Analyze the code, find the flaw SHORTCUT Get the answer from a source it wasn’t allowed to use SCORE PASS SCORE PASS WHAT IT SHOWS The capability the test was built to measure WHAT IT SHOWS Nothing about that capability The score alone can’t tell these apart. Inspect how each answer was obtained, and control what separate runs can share. In OpenAI’s account of its July 2026 Hugging Face incident, agents sought test solutions through unintended routes.
Both paths pass, so the score alone can’t separate them. Check how the answer was obtained, not just whether it was right. Download the image

Sources

Go deeper

From the conversation

Agent TrajectoryAI EvaluationCross-Model ReviewEval-Driven Development