RLHF (Reinforcement Learning from Human Feedback)

A pretrained model predicts likely text, which is not the same as text people find helpful, honest, or safe. Reinforcement learning from human feedback (RLHF) trains toward what people prefer. The approach was demonstrated in 2017 by Paul Christiano and colleagues at OpenAI and DeepMind, who trained agents on Atari games and simulated robot tasks from human comparisons between pairs of short clips, with feedback on less than 1% of the agent’s interactions (Christiano et al., Deep Reinforcement Learning from Human Preferences).

For language models the standard recipe comes from OpenAI’s InstructGPT paper (Ouyang et al., Training language models to follow instructions with human feedback). Labelers first write demonstrations used for supervised fine-tuning. They then rank several model outputs for the same prompt; those rankings train a reward model that predicts which output a person would prefer; and the language model is optimized with reinforcement learning to score well on that reward model. The headline result showed how much the method mattered: in human evaluations, outputs from a 1.3B-parameter InstructGPT model were preferred to outputs from the 175B-parameter GPT-3, a model with 100 times more parameters.

RLHF has known limits, catalogued by Stephen Casper and coauthors (Open Problems and Fundamental Limitations of RLHF). Human evaluators often disagree; the papers they survey report agreement rates of 63 to 77%. People cannot reliably judge outputs on hard tasks, so the model learns what looks right to a rater, and a single reward function cannot represent a diverse population’s preferences. They also note that deployed models fine-tuned with RLHF have given sycophantic responses and hallucinated, and that RLHF has not made models robust to jailbreaking. And a policy can learn to exploit the limits of human oversight; the paper’s example is a robotic hand, trained from 2D renderings to grasp an object, that learned to move into the line of sight between the evaluators and the object instead, because they could not tell the difference. The same proxy problem appears under reward hacking.

Newer methods change parts of the recipe. Direct preference optimization (DPO), for example, learns from the same kind of preference pairs with a simple classification loss, replacing the reinforcement learning stage (Rafailov et al., Direct Preference Optimization). The fine-tuning entry covers where these methods sit relative to supervised training. The model training topic collects the episodes on post-training.