In reinforcement learning, the policy is whatever decides what to do next. For a language model, the policy is the model itself: given a prompt, it chooses the next token. Policy optimization means sampling some answers, scoring them with a reward, and changing the model’s weights so that higher-scoring answers become more likely. A coding model, for example, writes ten attempts at a function, the ones that pass the tests get a reward, and the update makes passing code a little more likely next time.
Proximal Policy Optimization (PPO), published by OpenAI researchers in 2017, became the standard method. Its key idea is caution: PPO-Clip uses a clipped objective to discourage large changes in the policy, but it does not impose a strict bound on those changes. InstructGPT used PPO against a reward model, and it remains the textbook RLHF optimizer. Classic PPO also trains a second network, a value model, to estimate how good each situation is, which costs memory.
Group Relative Policy Optimization (GRPO), introduced in the DeepSeekMath paper in 2024, drops that value model. It samples a group of answers to the same prompt and scores each one against the group’s average, so an answer is rewarded for being better than its siblings. That makes it cheaper to run, and it suits tasks with checkable answers, which is why it is associated with reinforcement learning with verifiable rewards and reasoning models such as DeepSeek-R1. Policy optimization does not fix a bad reward: optimize hard against a flawed score and you get reward hacking.
In episode 43 at 23:53, Liquid AI’s Maxime Labonne says that for a chatbot, optimizing for preferences “is not optional,” and names “DPO, direct preference optimization, or more elaborate one like PPO.” DPO learns from preference pairs without a separate reward model or reinforcement learning loop; PPO is the more involved route.
Hear it from the guest
“So preference optimization techniques like DPO, direct preference optimization, or more elaborate one like PPO”
Quotes lightly edited to remove filler words.
Sources
- Schulman et al.: Proximal Policy Optimization Algorithms (2017) — Introduces PPO, a policy gradient method whose surrogate objective allows several epochs of minibatch updates on the same sampled data; checked October 24, 2026.
- OpenAI Spinning Up: Proximal Policy Optimization — Distinguishes PPO-Clip's clipped objective from a hard constraint and warns that the updated policy can still move too far; checked October 25, 2026.
- Shao et al.: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024) — Introduces Group Relative Policy Optimization (GRPO), a variant of PPO that improves mathematical reasoning while reducing PPO's memory use; checked October 24, 2026.
- DeepSeek-AI: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025) — Reports that reasoning abilities can be incentivized through pure reinforcement learning, the training run that made GRPO widely known; checked October 24, 2026.
- Hugging Face TRL: GRPO Trainer — Official library documentation showing GRPO training with reward functions that score groups of completions, including a check of answers against ground truth; checked October 24, 2026.
