People cannot rate millions of chatbot answers by hand, but they can rate a few thousand. A reward model stretches that effort. Labelers see two answers to the same question and pick the better one, a form of pairwise evaluation. A separate model is trained on those choices until it can predict which answer a person would prefer, and it outputs a single number for any new answer. During reinforcement learning, the main model generates answers, the reward model scores them, and policy optimization nudges the main model toward answers that score higher.
That is the core of RLHF, as used for InstructGPT and many later chat models. Some methods skip the separate model: DPO learns from the preference pairs directly, and reinforcement learning with verifiable rewards replaces the learned score with a check such as running unit tests or matching a math answer.
A reward model is only an approximation of what people want, and the model being trained will exploit any gap. If the reward model likes long, confident answers, training produces long, confident answers whether or not they are right. That failure is reward hacking, and it is why researchers evaluate reward models themselves with benchmarks such as RewardBench.
In episode 1 at 23:25, Galileo co-founder Atindriyo Sanyal describes the training stage “including RLHF, where you use human feedback as reward signal” and says much of the gain from GPT-3 to GPT-3.5 came from it. In episode 5 at 23:00, NVIDIA’s Vivienne Zhang points listeners to RewardBench as an open leaderboard. Galileo sponsored season one, and episode 1’s panel participants are Galileo’s co-founders.
Hear it from the guest
“Including RLHF, where you use human feedback as reward signal”
“RewardBench”
Quotes lightly edited to remove filler words.
Sources
- Christiano et al.: Deep reinforcement learning from human preferences (2017) — Learns a reward from non-expert human preferences between pairs of behavior clips and uses it to solve tasks without a hand-written reward function; checked October 24, 2026.
- Ouyang et al.: Training language models to follow instructions with human feedback (2022) — The InstructGPT recipe: collect labeler rankings of model outputs, train a reward model on them, then fine-tune the language model against it with PPO; checked October 24, 2026.
- Lambert et al.: RewardBench: Evaluating Reward Models for Language Modeling (2024) — Calls reward models the crux of RLHF and introduces a benchmark for evaluating them; checked October 24, 2026.
