RLHF vs RLVR vs RLAIF: How AI models learn

Pretraining teaches a model to predict the next token on the internet. That skill produces fluent text, but it does not tell the model which answer a person would prefer, which reasoning trace is correct, or which refusal keeps a user safe. The gap shows up the first time you ship. The base model hallucinates a citation, writes insecure code that compiles, or answers a harmful request with polite evasion.

Post-training closes that gap with reinforcement learning. RLHF, RLAIF, and RLVR are three ways to supply the reward that drives that RL loop. They differ in who judges the output and what counts as correct. Frontier labs now combine several of them in sequence. Understanding when each one applies explains why models improved on reasoning in 2024 and 2025, why alignment still costs real human effort, and where each method fails.

The shared loop underneath all three

All three methods build on the same three-stage pattern described in InstructGPT and Anthropic's helpful and harmless work.

1. Start from a pretrained base. Fine-tune it on demonstrations to create a supervised policy that follows instructions.
2. Collect judgments about which output is better for a given prompt. Train a reward model to predict those judgments.
3. Optimize the policy against the reward model with a reinforcement learning algorithm, usually Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO), with a KL penalty that keeps the new policy close to its starting point.

InstructGPT reported that a 1.3B parameter model tuned with this loop was preferred by humans over the 175B GPT-3 base, while Anthropic showed the same loop could improve almost all NLP evaluations when trained on both helpfulness and harmlessness comparisons.

The difference between RLHF, RLAIF, and RLVR lives entirely in step 2: the source of the judgment and the definition of better.

RLHF: learning taste and safety from people

Reinforcement learning from human feedback is the oldest of the three outside of games. Christiano et al. introduced it for control tasks in 2017, Stiennon et al. applied it to summarization in 2020, and Ouyang et al. and Bai et al. made it the standard post-training step for language assistants in 2022.

How it works:

• Workers see a prompt and two model completions side by side and choose the one they prefer. For harmlessness data, the task is inverted and workers choose the more harmful response to build a contrasting signal.
• Those pairs train a preference model that outputs a scalar score. The loss pushes the score of the preferred completion above the rejected one.
• The policy generates new responses to fresh prompts. The reward model scores them, and PPO updates the policy to increase scores. A per-token KL penalty against the supervised model prevents the policy from drifting into high-scoring gibberish.
• Labs iterate online, collecting new comparisons on the latest policy each week and retraining both reward model and policy.

What it teaches well:

• Tone, helpfulness, honesty calibrations, and refusal style where no checker can decide correctness. A verification script cannot tell you whether an apology sounds sincere or whether a summary respects the nuance of a request. Human judgment can.
• Open-ended instruction following across thousands of diverse prompts. InstructGPT used labeler-written demos plus prompts from API users to cover that distribution.

Where it struggles:

• Scale and cost. Ouyang et al. used 40 contractors and multiple rounds of labeling. Good judgments require training, and even trained raters disagree. Quality control matters as much as quantity.
• Noisy grounding. When tasks need factual or mathematical correctness, human raters often cannot verify the answer. They reward confident, well-formatted wrong answers. That reward hacking is the central failure mode of RLHF. Later work shows proxy reward and true quality can diverge once reward exceeds a threshold, and proxy scores keep climbing while human win rate falls.
• Trade-off between helpful and harmless. Anthropic found that a reward model trained mostly on helpfulness performs far below chance at predicting harmlessness. A joint mix is needed, which means careful balancing of datasets that often pull in opposite directions.

RLHF remains essential for behavior where correctness has no formal definition.

RLAIF: scaling the same judgment with an AI judge

Reinforcement learning from AI feedback replaces the human ranker with an off-the-shelf language model prompted to choose the better completion. Anthropic named the pattern in Constitutional AI in December 2022, and Lee et al. measured it directly against RLHF in the RLAIF paper.

Constitutional AI has two phases that illustrate the mechanics clearly. In the supervised phase, the model samples a response, critiques its own output against a written constitution of principles, revises the response, and is then fine-tuned on the revised answer. In the RL phase, the fine-tuned model generates two samples, a feedback model judges which aligns better with the constitution, those AI preferences train a preference model, and PPO uses that model as the reward.

Lee et al. tested the simpler case without a constitution on three tasks. Human evaluators preferred RLAIF over a supervised baseline 71% of the time on summarization and 63% on helpful dialogue. RLHF numbers were 73% and 64% on the same tasks, a difference that was not statistically significant. On harmlessness, RLAIF produced a higher harmless rate than RLHF, 88% versus 76%, both above the 64% baseline. A variant that skipped the reward model distillation and asked the LLM for a direct scalar reward during RL performed even better, with a 74% win rate over the baseline and a 60% head-to-head win over the distilled version using the same labeler.

Two properties matter for readers evaluating the method:

• The labeler can be the same model that is learning. Lee et al. improved a policy even when the judge was the same size as the policy, and in some experiments the same checkpoint. Self-improvement is possible because the model judges better as a pairwise ranker than it generates as a single sampler.
• The ceiling is the judge. RLAIF copies the judge's preferences, including its biases and blind spots. Anthropic reported a Pareto improvement where the constitutional RLAIF model was both more helpful and more harmless than RLHF, but that result required a carefully written constitution and chain-of-thought reasoning during judgment. Without that, an AI judge that prefers verbosity, apologies, or hedged answers will teach the policy to produce them.

Use RLAIF when you have a strong judge model and a clear set of principles, and when human labeling would be too slow, too expensive, or too exposed to harmful content to scale. Use human judgments when nuance, stakeholder legitimacy, or safety certification requires a human in the scoring definition.

RLVR: correctness where a checker can decide

Reinforcement learning with verifiable rewards, now often shortened to RLVR, gives reward only when an answer can be checked automatically. It became the centerpiece of the reasoning push in 2024 and 2025.

The idea is blunt: if you can verify the answer, do not model human preference. Score 1 for correct, 0 for wrong, and let RL find the thinking that earns the 1.

DeepSeek-R1 is the most documented example. The team built on DeepSeek-V3 Base and used GRPO as the optimizer. For DeepSeek-R1-Zero they skipped supervised fine-tuning entirely and applied RL directly to the base model. The reward was rule based: accuracy rewards that check whether a math answer matches the boxed format or whether code passes compiler tests, plus a format reward that requires the chain of thought to sit between thinking tags. No neural reward model judged the trace itself. The model generated longer traces that included verification and reflection, and AIME 2024 pass at one rose from 15.6% to 71% and to 86.7% with majority voting. The zero model was strong but mixed languages and was hard to read, so DeepSeek-R1 added a cold start of thousands of human-curated long traces, ran reasoning-focused RL again with a language consistency reward, did rejection sampling to keep only correct traces, fine-tuned on 600k mixed reasoning and non-reasoning examples, and then did a second RL stage that mixed rule based and model based rewards.

OpenAI described a similar shift for the o1 series. OpenAI trained o1 with large scale RL that teaches the model to refine its chain of thought, correct mistakes, and try alternative strategies. Performance improved with added RL at train time and with added thinking time at test time. The team also noted that instruction on safety policies lands better when it is learned inside that chain of thought, and that monitoring the trace gives a new lever for alignment.

The technical detail that enables RLVR at scale is GRPO. Unlike PPO, which learns a value function as a critic, GRPO samples a group of outputs for the same prompt and normalizes rewards within the group to form the advantage. That removes the critic memory cost, handles the binary 0 or 1 reward cleanly, and yields an adaptive correction: when most samples in a group are wrong, a correct sample gets a large positive advantage, and when most are correct, the remaining wrong samples are strongly penalized. Research on verifiable rewards frames it as a KL-regularized contrastive loss whose optimum provably lifts success rate above the reference policy when regularization is set properly.

Strengths:

• Resilient to preference hacking. A model cannot talk its way to a higher score. It either passes the test or it does not.
• Efficient at discovery. Because the reward does not encode a human style template, RL can explore reasoning patterns beyond what humans demonstrated, including verification and backtracking that emerge during training.

Limits:

• Narrow domain. RLVR applies where verification is cheap and reliable: math with exact answers, code with unit tests, logic puzzles, and some STEM problems. Open writing, product advice, or medical triage have no verifier.
• Verifier quality still governs outcome. A weak test suite, a leaky boxed answer extractor, or a generative judge used as verifier can be hacked like any reward model. Studies on GRPO with verifiable rewards show that with low KL regularization the policy still learns to exploit the reward by emitting short, barely formatted answers that score 1 without useful reasoning. Token-level losses, format rewards, and adequate group sizes reduce that behavior but do not remove the need for a strong verifier.
• Broad skills degrade without mixing. The DeepSeek-R1-Zero stage trained only on verifiable rewards and underperformed on writing and open QA. The final model recovered general helpfulness only after supervised data from other domains and a second RL stage with a model based reward.

RLVR is therefore not a replacement for preference based RL. It is a specialist stage for domains with ground truth.

How the three compare

Dimension	RLHF	RLAIF	RLVR
Who scores	Trained human raters	A language model prompted with principles or a direct preference question	A program or extractor: math checker, test harness, format validator
Signal type	Pairwise preference	Pairwise preference or direct scalar from the AI judge	Binary or scalar verifiable outcome
What it teaches	Taste, helpfulness, tone, and nuanced safety calls	The same taste and safety judgments, scaled	Correctness, multi-step reasoning, tool use where outcome is testable
Scales to	Limited by rater hiring and agreement	Limited by judge model capability and inference budget	Limited by availability of high quality verifiable prompts and trustworthy checkers
Main failure mode	Reward hacking on unverifiable tasks, reward model overoptimization	Inherits judge biases, can amplify preference for fluent but wrong answers	Overfitting to verifier quirks, struggles outside verifiable domains, short-circuiting of reasoning when KL is weak
Evidence for parity	InstructGPT human win rates over 175B GPT-3 with 100x fewer parameters	RLAIF 71% vs RLHF 73% on summarization, no significant difference; harmless rate 88% vs 76%	DeepSeek-R1 Zero AIME 15.6% to 71% via pure RL without demos; o1 reasoning gains scaling with RL compute

How labs sequence them

Frontier pipelines no longer choose a single method. They chain them in the order that limits each method's weakness.

A pattern visible in 2025:

1. Cold start with supervised fine-tuning on a small set of high quality demonstrations that show readable chain of thought. This stabilizes early RL and controls language mixing.
2. Reasoning focused RL with verifiable rewards on math, code, and logic. This builds the core reasoning trace and lifts success probability in those domains.
3. Rejection sampling and another supervised phase on the best rollouts, now mixed with general writing and factual data, to restore breadth and readability. DeepSeek-R1 collected about 600k reasoning samples this way plus non-reasoning corpora.
4. General RL with a mix of rule based rewards for reasoning and preference based rewards for helpfulness and harmlessness. DeepSeek-R1 and o1 both describe this final stage as a blend. Anthropic does a similar blend in Constitutional AI by using AI feedback guided by a constitution for harmlessness after the core capability training.

The sequence explains the trade-off a product team feels. Reasoning improves without human demos, but deployability still needs taste and safety tuning from preference methods.

What this means for teams that use these models

Most SaaS teams will not train with any of these methods directly. They will choose, route between, and evaluate models trained with them. Three decisions follow from the differences above.

1. Match the benchmark to the reward behind it. A model that led on SWE-bench Pro or AIME likely benefited from RLVR on code or math. A model that feels better in open-ended conversation or refusal style likely benefited from RLHF or RLAIF. Do not expect excellence on unverifiable writing from a model whose gains came only from verifiable rewards.
2. Evaluate verifiability in your own product. If your agent produces an answer that a function can check, such as generating a query that must run, a reconciliation that must balance, or code that must pass tests, you can use RLVR-style checkers as your own evals. If correctness is defined as matching a domain expert's judgment, preference based evals remain necessary. Build both.
3. Plan for guardrails that match the failure mode. Preference based models fail by being agreeably wrong. Verifiable reward models fail by gaming a weak verifier or collapsing to minimal correct answers. Monitor for both. Check citations and tool outputs even when the thinking looks detailed.

Foundation model training of this kind sits outside embedded product work. When the topic concerns agents inside a SaaS product, the relevant integration is an agent that lives in the product, reasons over product records, and calls scoped tools under the customer's permission model, or a managed MCP server that exposes selected product capabilities to an external client. Those decisions are separate from how the underlying model was rewarded during its own training, but they determine whether reasoning strength translates into product work that completes safely.

Sources

• Ouyang et al., Training language models to follow instructions with human feedback, arXiv 2203.02155, 2022. Three-stage SFT, reward model, and PPO with KL penalty; InstructGPT results.
• Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, arXiv 2204.05862, 2022. Helpful and harmless reward modeling and the trade-off between them.
• Bai et al., Constitutional AI: Harmlessness from AI Feedback, arXiv 2212.08073, 2022. Supervised critique-revision and RL from AI feedback guided by a constitution.
• Lee et al., RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback, arXiv 2309.00267, ICML 2024. Direct measurement showing RLAIF 71% vs RLHF 73% on summarization and harmless rate 88% vs 76%.
• OpenAI, Learning to reason with LLMs, September 12, 2024. o1 trained with large scale RL for chain-of-thought reasoning and deliberative alignment.
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv 2501.12948, 2025 and Nature 645, September 17, 2025. R1-Zero without SFT, rule based accuracy and format rewards, GRPO, staged pipeline with cold start, rejection sampling, and secondary RL.
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv 2402.03300, 2024. GRPO definition, group-normalized advantage without a critic.
• Gao et al., Scaling Laws for Reward Model Overoptimization, arXiv 2210.10760, 2022, and subsequent reward hacking analyses showing proxy reward and win rate divergence and the role of KL bounds.