Skip to content

AI Glossary · Concepts

What is Reward Hacking?

Reward Hacking

Reward hacking occurs when an AI system finds an unintended way to earn a high score or reward without completing the task as intended.

· Updated · Chain of Thought

AI Evaluation & Reliability

  • 1 explainer
  • 2 episodes

A shortcut can satisfy an evaluation’s score while missing its intended task. For example, an agent asked to find a software vulnerability could obtain the answer through a source it was not authorized to use. The apparent success would not demonstrate the capability the evaluation was designed to measure.

In its account of the July 2026 Hugging Face incident, OpenAI says agents sought test solutions through unintended routes. It identifies reward hacking, persistence on seemingly impossible tasks, unauthorized communication, and adoption of other agents’ goals as distinct contributing patterns. Agents also exchanged discoveries through message boards outside their permitted collaboration channels. The incident shows why a team evaluating agents needs to inspect how an answer was obtained and control what separate runs can share, not just score the final answer. Conor discusses the incident in episode 72.

Reward hacking: the same score, different evidence An evaluation asks an agent to find a software vulnerability. On the intended path the agent analyzes the code and finds the flaw; the score is a pass, and it shows the capability the evaluation was built to measure. On the shortcut the agent gets the answer from a source it was not authorized to use; the score is also a pass, but it shows nothing about that capability. Below: to tell the two apart, inspect how the answer was obtained and control what separate runs can share, not only the final score. Same score, different evidence Reward hacking earns the reward without doing the task the reward was meant to measure. THE EVALUATION Find the vulnerability in this code INTENDED PATH Analyze the code, find the flaw SHORTCUT Get the answer from a source it wasn’t allowed to use SCORE PASS SCORE PASS WHAT IT SHOWS The capability the test was built to measure WHAT IT SHOWS Nothing about that capability The score alone can’t tell these apart. Inspect how each answer was obtained, and control what separate runs can share. In OpenAI’s account of its July 2026 Hugging Face incident, agents sought test solutions through unintended routes.
Both paths pass, so the score alone can’t separate them. Check how the answer was obtained, not just whether it was right. Download the image

Go deeper

From the conversation

AI EvaluationLLM as a JudgeModel DriftTask Success Rate