Chapter 4.5 β€” From Base Model to Assistant

Contents

  1. What a pretrained model is missing
  2. Supervised fine-tuning: teaching the format
  3. FLAN and the case for multi-task instruction tuning
  4. Reward models and preference data
  5. RLHF: optimizing the policy against a learned reward
  6. InstructGPT: making the full pipeline concrete
  7. DPO: collapsing preference optimization into a single loss
  8. Cheaper alternatives: rejection sampling and GRPO
  9. Scaling preference data with AI feedback
  10. Interview angle
  11. Self-check questions
  12. Sources

1. What a pretrained model is missing

Chapter 4.4 dealt with the purely infrastructural problem of getting a large model trained at all; this chapter picks up with a model that has cleared that hurdle β€” it has been pretrained, at whatever scale and token budget the previous two chapters would recommend, on the causal next-token-prediction objective from Chapter 1.1. Such a model is, in a real and well-earned sense, extremely good at one thing: continuing a piece of text plausibly, in whatever style and register the preceding context implies. That is a genuinely powerful capability, and it is not the same capability as "being a helpful conversational assistant." A raw pretrained model given the prompt "How do I center a div in CSS?" is just as likely to continue it with a list of five other CSS questions someone might plausibly ask next, because that is a perfectly plausible continuation of text that looks like a forum post, as it is to actually answer the question. The model has learned the statistics of text as it appears on the internet, and "answer the question helpfully and stop" is only one of many patterns present in that distribution, not the dominant one.

Turning such a model into something that reliably behaves like an assistant β€” answering the question that was actually asked, in a helpful register, stopping when it's done, refusing requests it shouldn't fulfill β€” requires additional training stages specifically designed to narrow the model's behavior toward that one pattern, deliberately, rather than hoping it emerges as a side effect of scale. This chapter covers the two dominant stages of that process as it has developed: supervised fine-tuning, which teaches the conversational format directly, and preference optimization (RLHF and its more recent alternatives), which further shapes the model's behavior using comparative human or AI judgments about which of several candidate responses is better.

2. Supervised fine-tuning: teaching the format

The first and most direct step is supervised fine-tuning, often abbreviated SFT and also called instruction tuning: take the pretrained model and continue training it, using the same underlying next-token-prediction loss, but now on a curated dataset of (instruction, response) pairs written or selected specifically to demonstrate the format and behavior you want β€” a question or task on one side, and a high-quality, helpful, appropriately-scoped response on the other. Because the loss is still just next-token prediction, no new architecture or objective is required; what's new is entirely the data. Given enough examples of well-formed instruction-response pairs, the model's distribution over plausible continuations shifts substantially toward "answer this helpfully and then stop" simply because that pattern now dominates the fine-tuning data, even though it was only one pattern among many in the vastly larger pretraining corpus.

It's worth being precise about what SFT does and does not accomplish, because this is a frequent point of confusion. SFT teaches format and surface behavior extremely effectively β€” it is why fine-tuned models reliably produce assistant-shaped responses rather than forum-post-shaped continuations β€” but it is comparatively data-hungry to use as the sole mechanism for teaching subtler behavioral judgments, like which of two roughly-correct-but-differently-toned responses is actually preferred, because writing enough labeled demonstrations to cover every such distinction at scale is expensive and doesn't scale as well as comparative judgments do. This limitation is exactly what motivates the preference-optimization stage described in the rest of this chapter.

3. FLAN and the case for multi-task instruction tuning

Wei et al.'s FLAN work provided an important early demonstration that instruction tuning's benefits aren't confined to the specific tasks you happened to include in the fine-tuning set. They fine-tuned a pretrained language model on a large collection of tasks phrased as natural-language instructions, spanning many different task types, and found that the resulting model generalized substantially better to entirely new, unseen tasks at zero-shot evaluation time than the pretrained model did without this instruction tuning, and often better than task-specific fine-tuning on individual tasks in isolation. The key finding worth stating precisely is that diversity of instruction-formatted tasks during fine-tuning, rather than volume of examples for any single task, is what drove the generalization improvement β€” the model appeared to be learning something more general about "how to follow an instruction," transferable across task types, rather than memorizing narrow task-specific patterns. This result is part of why instruction tuning is now treated as a standard, near-universal stage of turning a pretrained model into a usable assistant, rather than an optional add-on for specific use cases.

4. Reward models and preference data

SFT gets a model to produce well-formed, helpful-shaped responses, but it doesn't give you a mechanism for expressing graded preferences between responses that are both plausible but not equally good β€” and collecting that kind of graded, comparative judgment turns out to be both more natural for humans to provide and more information-dense per unit of labeling effort than writing full demonstrations from scratch. It's much easier and more reliable for a human rater to look at two candidate responses to the same prompt and say which one they prefer than it is for them to write, from nothing, the single ideal response β€” comparative judgments are cheaper to elicit and less prone to inconsistency than absolute ratings or from-scratch demonstrations.

This observation motivates training a separate reward model: collect a dataset of prompts, each paired with several candidate model-generated responses, have human raters indicate which of a pair of candidates they prefer, and train a model β€” usually initialized from the same pretrained/fine-tuned base β€” to predict a scalar score for a response such that its scores are consistent with the observed pairwise preferences. This is typically framed as a classification-style loss over pairs, following the Bradley-Terry-style formulation used in Christiano et al.'s foundational work on learning from human preferences: the reward model is trained so that the probability it assigns to one response being preferred over another, derived from the difference in their predicted scalar rewards via a sigmoid, matches the empirical human preference frequency. Once trained, this reward model can score any candidate response to any prompt, giving you a differentiable-in-effect, automatically computable stand-in for "how good is this response," which is exactly the signal needed to optimize the language model itself against, at a scale no amount of direct human labeling of every generated response could support.

Concretely: let $y_w$ denote the response a rater preferred ("winning") and $y_l$ the one they didn't ("losing"), for the same prompt $x$. The Bradley-Terry model says the probability of that observed preference is a sigmoid of the reward difference, $$P(y_w \succ y_l \mid x) = \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big),$$ and the reward model is trained to maximize the log-likelihood of the observed preferences, i.e. to minimize $$\mathcal{L}_R(\theta) = -\,\mathbb{E}_{(x,y_w,y_l)\sim D}\Big[\log \sigma\big(r_\theta(x,y_w) - r_\theta(x,y_l)\big)\Big].$$

5. RLHF: optimizing the policy against a learned reward

Reinforcement Learning from Human Feedback, RLHF, is the procedure that takes this learned reward model and uses it to further train the language model β€” now treated as a policy in the reinforcement learning sense, whose "actions" are the tokens it generates β€” to produce responses that score highly under the reward model. Christiano et al.'s original formulation of learning policies from human preference comparisons, developed initially outside the language modeling context, established the core idea that a reward model trained from pairwise human preferences can substitute for a hand-specified reward function in a standard reinforcement learning loop; applying this to language models means using the reward model's score on a generated response as the reward signal for training the language model via policy gradient methods, most commonly Proximal Policy Optimization, PPO, described by Schulman et al., which stabilizes policy gradient training by constraining how much the policy is allowed to change in a single update relative to the policy that generated the training data, avoiding the destructively large policy updates that plain policy gradient methods are prone to.

PPO's stabilization mechanism has a concrete form. Writing the probability ratio between the new and old policy for a sampled token as $r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$ and $\hat A_t$ for the estimated advantage of that token, PPO maximizes a clipped surrogate objective $$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(r_t(\theta)\,\hat A_t,\ \operatorname{clip}(r_t(\theta),\, 1-\epsilon,\, 1+\epsilon)\,\hat A_t\big)\Big],$$ which takes the smaller of the unclipped and clipped estimates, so that a policy update which would push the ratio too far from $1$ β€” in whichever direction currently helps the objective β€” stops contributing extra gradient signal past that point. That's the mechanical reason PPO can't run away from the old policy in a single update.

A critical, easy-to-underweight detail in this loop is the KL penalty: the training objective doesn't just maximize the reward model's score, it maximizes reward minus a penalty term proportional to the KL divergence between the current policy and the original SFT policy it started from. Written out, the full RLHF objective being optimized is $$\max_\pi\; \mathbb{E}_{x\sim D,\ y\sim \pi(\cdot\mid x)}\big[r_\theta(x,y)\big] \;-\; \beta\, D_{\mathrm{KL}}\big[\pi(\cdot\mid x)\,\|\,\pi_{\mathrm{SFT}}(\cdot\mid x)\big],$$ where $\beta$ controls how strongly the penalty is weighted against raw reward. This penalty exists because optimizing purely against a learned reward model, with no constraint at all, reliably produces reward hacking β€” the policy discovers ways to generate text that scores very highly under the imperfect, learned reward model's approximation of "good response" without actually being good by the standards the reward model was trained to approximate, since the reward model is itself only an approximation trained on a finite, imperfect sample of human judgments and is exploitable if you optimize against it hard enough. The KL penalty keeps the policy anchored close enough to the original SFT model's distribution that it can't drift into these degenerate, reward-model-exploiting regions, trading off some potential reward-model score for staying within a region where the reward model's judgments are more likely to still be trustworthy.

6. InstructGPT: making the full pipeline concrete

Ouyang et al.'s InstructGPT paper is the work that took this three-stage pipeline β€” supervised fine-tuning, reward model training from human preference comparisons, and PPO-based RLHF against that reward model with a KL penalty against the SFT policy β€” and demonstrated it concretely and convincingly at meaningful scale, showing that the resulting model was strongly preferred by human evaluators over the base pretrained model and even over much larger models that hadn't been through this pipeline, on instruction-following and helpfulness criteria specifically. This paper is worth knowing in real detail rather than just by name in an interview context, because it is the reference point essentially every subsequent discussion of "the RLHF pipeline" assumes as background, and because it made concrete and public exactly the pipeline described in the two preceding sections, which had previously existed in a more scattered, less unified form across the preference-learning and RL literature.

7. DPO: collapsing preference optimization into a single loss

RLHF as described above works, but it is operationally heavy: it requires training and maintaining a separate reward model, and it requires running an actual reinforcement learning loop β€” sampling responses from the current policy, scoring them, and applying PPO updates β€” which is notoriously more finicky to get stable and reproducible than ordinary supervised training, with more hyperparameters, more moving pieces, and more failure modes than a straightforward gradient-descent fine-tuning run.

Direct Preference Optimization, DPO, introduced by Rafailov et al., offers a way to skip most of that machinery while pursuing essentially the same underlying goal. The key mathematical move is to show that the RLHF objective β€” maximize expected reward under the learned reward model subject to a KL penalty against a reference policy β€” has a closed-form relationship between the optimal policy and the reward function it's implicitly optimizing against, which lets you reparameterize the reward function directly in terms of the policy itself. Substituting this reparameterization back into the reward-model training loss (the same Bradley-Terry pairwise preference loss described earlier) produces a loss that can be optimized directly on the language model's own parameters, using nothing but a dataset of preference pairs and simple supervised-style gradient descent, with no separate reward model and no reinforcement learning loop at all. Concretely, that loss is $$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\,\mathbb{E}_{(x,y_w,y_l)\sim D}\left[\log \sigma\!\left(\beta \log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)} \;-\; \beta \log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right],$$ which is just the Bradley-Terry loss from section 4 with the reward difference rewritten in terms of the policy's own log-probability ratios against the reference model β€” no reward model anywhere in sight. Rafailov et al. show that this loss, under reasonable assumptions, optimizes the same underlying objective that RLHF does β€” the same KL-constrained reward maximization β€” just via a different, more direct and more stable optimization path. This is a genuinely elegant piece of derivation worth being able to sketch in an interview: DPO isn't a heuristic approximation to RLHF, it's a reformulation that arrives at the same target via ordinary supervised-style optimization, which is why it has become a popular alternative wherever the operational complexity of a full RL loop isn't worth its cost.

8. Cheaper alternatives: rejection sampling and GRPO

DPO removes the RL loop entirely, but it's worth knowing two other ways teams cut cost out of this pipeline, because interviewers use them to check whether you think of RLHF as a fixed recipe or as one point on a genuine cost-versus-quality spectrum.

The cheapest option skips reinforcement learning altogether: rejection sampling, sometimes called best-of-$n$ fine-tuning, samples several candidate responses to each prompt from the current SFT model, scores every candidate with the reward model from section 4, and then fine-tunes on only the highest-scoring response per prompt using the same ordinary next-token-prediction loss SFT already uses. Formally, the fine-tuning target for each prompt is just the arg max of the reward model's scores over the sampled candidates, $$y^* = \arg\max_{i \in \{1,\dots,n\}} r_\theta(x, y_i),$$ with the model then fine-tuned toward $y^*$ using ordinary next-token-prediction loss. There is no policy-gradient machinery at all β€” the reward model's only job is to pick which of the already-generated candidates to imitate, so the update is exactly as stable as supervised fine-tuning, because it is supervised fine-tuning, just on self-generated, reward-model-filtered data rather than human-written demonstrations. Llama 2's published training recipe used exactly this technique, alongside PPO, as an earlier and cheaper stage of preference optimization. Best-of-$n$ resampling at inference time β€” generate several candidates, use the reward model to pick the best one before showing it to the user β€” is the same idea applied at serving time rather than training time, and it's worth naming explicitly in an interview: any training-time preference-optimization method has to be worth its cost relative to this comparatively trivial inference-time baseline, or there's no reason to prefer it.

GRPO, Group Relative Policy Optimization, introduced by Shao et al. as part of the DeepSeekMath work, cuts cost out of the RL side specifically rather than replacing it. Standard PPO needs a separate value network β€” a learned critic, typically about the same size as the policy itself β€” to estimate the baseline that turns a raw reward into a lower-variance advantage signal, roughly doubling the memory and compute needed for the RL step. GRPO removes the critic by sampling a group of several completions for the same prompt and using that group's own mean reward as the baseline: a completion's advantage is just how much better or worse it scored than the average of its own group, normalized by the group's standard deviation, with no separate network learning to predict that baseline at all. Formally, for a group of $G$ completions sampled from the same prompt with rewards $r_1,\dots,r_G$, completion $i$'s advantage is $$A_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)},$$ substituted directly for $\hat A_t$ in the same clipped surrogate objective PPO uses in section 5 β€” with one further difference: GRPO adds its KL penalty against the reference policy as a separate term in the loss itself, rather than folding it into the reward the way RLHF's objective in section 5 does. This is cheaper per step, simpler to keep stable, and β€” as later chapters cover β€” turns out to compose particularly well with objectives that have a clean, verifiable reward (a checkable final answer, a passing test) rather than a learned reward model's noisier scalar, which is exactly the setting Chapter 5.1's discussion of DeepSeek-R1 picks up. The throughline worth taking from this section is that the "RLHF pipeline" from sections 4 through 6 is one point among several on a real spectrum trading operational complexity for how directly the method optimizes what you actually care about β€” rejection sampling nearest the simple end, full PPO nearest the expensive end, and DPO and GRPO each cutting a different specific piece of that expense while keeping most of the benefit.

9. Scaling preference data with AI feedback

Everything described so far assumes a supply of human-labeled preference comparisons, and that supply is itself a bottleneck: human labeling is slow, expensive, and hard to scale to the volume of comparisons a large preference-optimization run can productively use. RLAIF, described by Lee et al., addresses this bottleneck by substituting AI-generated preference labels β€” having a separate language model judge which of two candidate responses is better, according to some specified criteria β€” for at least part of the human-labeled preference data the pipeline would otherwise require, and shows that preference optimization using these AI-generated labels can achieve comparable results to using human-labeled preferences, at a fraction of the labeling cost and with far greater scalability. This idea, using model-generated judgments to substitute for or supplement human ones in the preference-data pipeline, is closely related to the Constitutional AI approach covered in full in the next chapter, which develops a specific, principled version of this same general strategy for the particular case of harmlessness and safety-relevant preferences rather than general helpfulness.

Everything covered in this chapter β€” SFT, RLHF, DPO, and AI-assisted preference data β€” is aimed at making a model helpful and well-behaved in the ordinary sense of following instructions competently and producing responses people prefer. It says comparatively little, so far, about a distinct and harder problem: making sure the model behaves acceptably even under adversarial pressure, unusual or edge-case inputs, and requests specifically designed to elicit harmful behavior, which is the subject the next chapter takes up directly. That chapter is not quite the last stop in this part, though: after alignment and safety are covered, Chapter 4.7 turns to a more narrowly practical concern β€” how to fine-tune a model at all when you can't afford to update its full set of parameters β€” before this part hands off to Part V.

10. Interview angle

Q: What specifically does supervised fine-tuning accomplish that pretraining alone does not, and what does it fail to accomplish on its own? A strong answer states that SFT shifts the model's output distribution toward assistant-shaped, instruction-following responses by training on curated (instruction, response) demonstrations with the same next-token loss used in pretraining, but it's comparatively poor at teaching fine-grained comparative judgments (which of two decent responses is better) because writing enough demonstrations to cover every such distinction doesn't scale β€” this is exactly the gap preference optimization fills.

Q: Why train a separate reward model instead of directly optimizing the language model against human ratings? A strong answer explains that collecting a scalar human rating for every candidate response at the volume needed for RL-style optimization isn't practical, whereas pairwise comparisons are cheap and reliable to collect from humans; training a reward model on those pairwise comparisons (via a Bradley-Terry-style loss) produces an automatically computable proxy for "how good is this response" that can be queried as often as needed during the policy optimization loop, which real-time human labeling could never support.

Q: Why does RLHF need a KL penalty against the original SFT policy, and what specifically goes wrong without it? A strong answer names reward hacking directly: without a constraint, the policy can drift toward regions of output space that score artificially high under the imperfect, learned reward model without actually being good by the standards the reward model was meant to approximate, since the reward model is only an approximation trained on finite data and is exploitable. The KL penalty keeps the policy anchored near the SFT model's distribution, where the reward model's judgments remain more trustworthy, at the cost of some potential reward-model score.

Q: How does DPO avoid needing a separate reward model or an RL loop, and is it really solving the same problem as RLHF? A strong answer sketches the reparameterization: the KL-constrained reward maximization objective has a closed-form relation between the optimal policy and its implicit reward function, so substituting that relation into the reward-model training loss yields a loss expressed directly in terms of the policy's own parameters, optimizable via ordinary supervised gradient descent on preference pairs. It should also state clearly that this isn't a heuristic shortcut β€” under the paper's assumptions it targets the same underlying KL-constrained objective as RLHF, just via a more stable optimization path.

Q: What does FLAN's result about instruction tuning generalization actually show, and why does it matter for how you'd design an instruction-tuning dataset? A strong answer states that generalization to unseen tasks improved with the diversity of instruction-formatted tasks in fine-tuning, not simply the volume of examples for any one task, implying that a well-designed instruction-tuning dataset should prioritize broad task-type coverage over deep coverage of any narrow task if the goal is generalization to new instructions at deployment time.

Q: Why bother with PPO at all if rejection sampling (best-of-$n$ fine-tuning) is so much simpler and more stable? A strong answer notes that rejection sampling can only select among responses the current policy already assigns some probability to β€” it can improve the policy toward its own best outputs, but it can't discover a genuinely better response strategy the way an RL update, which adjusts the policy's parameters directly based on the reward signal, in principle can. In practice teams often use rejection sampling as a cheaper first stage and reserve full RL (or DPO) for squeezing out further gains, rather than treating either as strictly superior in isolation.

Q: What specifically does GRPO remove from PPO, and what does it cost in exchange? A strong answer identifies the value network: PPO trains a separate critic to estimate a baseline for the advantage calculation, roughly doubling the compute and memory of the RL step; GRPO replaces that learned baseline with the empirical mean reward of a sampled group of completions to the same prompt, normalized by the group's standard deviation. The cost is that the advantage estimate is noisier for any single prompt unless you sample a reasonably sized group, trading per-step compute for a somewhat higher-variance signal.

11. Self-check questions

  1. Why is "continues text plausibly" a meaningfully different capability from "answers a question helpfully and stops," even though both can emerge from a next-token-prediction objective?
  2. What loss function does supervised fine-tuning actually use, and what specifically changes relative to pretraining?
  3. What did FLAN's experiments suggest was the actual driver of improved zero-shot generalization after instruction tuning?
  4. Why are pairwise comparisons a more practical form of human feedback to collect at scale than either scalar ratings or full written demonstrations?
  5. Explain reward hacking in the RLHF context and the specific mechanism (the KL penalty) used to mitigate it.
  6. What is the key mathematical move that lets DPO replace a reward model and an RL loop with a single supervised-style loss?
  7. What bottleneck does RLAIF address, and what's the basic mechanism it uses to address it?
  8. Why is rejection sampling fine-tuning exactly as stable as ordinary supervised fine-tuning, when full RLHF is notoriously not?
  9. What specific component does GRPO eliminate relative to PPO, and what does it substitute in its place to estimate an advantage?

12. Sources

  • Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback (InstructGPT). NeurIPS 2022. arXiv:2203.02155. https://arxiv.org/abs/2203.02155
  • Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290. https://arxiv.org/abs/2305.18290
  • Christiano, P., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS 2017. arXiv:1706.03741. https://arxiv.org/abs/1706.03741
  • Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. https://arxiv.org/abs/1707.06347
  • Wei, J., Bosma, M., Zhao, V. Y., et al. (2022). Finetuned Language Models Are Zero-Shot Learners (FLAN). ICLR 2022. arXiv:2109.01652. https://arxiv.org/abs/2109.01652
  • Lee, H., Phatale, S., Mansoor, H., et al. (2023). RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267. https://arxiv.org/abs/2309.00267
  • Touvron, H., Martin, L., Stone, K., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. https://arxiv.org/abs/2307.09288
  • Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). arXiv:2402.03300. https://arxiv.org/abs/2402.03300

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

What does supervised fine-tuning (SFT) actually change relative to pretraining?

Explanation: No new architecture or objective is needed β€” only the data changes, shifting the model's output distribution toward assistant-shaped, instruction-following responses.

According to FLAN's results, what actually drove improved zero-shot generalization after instruction tuning?

Explanation: The model appeared to learn something general about "how to follow an instruction," transferable across task types, rather than memorizing narrow task-specific patterns.

Why does RLHF include a KL penalty against the original SFT policy?

Explanation: The reward model is only an approximation trained on finite human judgments and is exploitable β€” the KL penalty keeps the policy anchored near the SFT distribution where the reward model stays trustworthy.

What is the key mathematical move that lets DPO replace a separate reward model and an RL loop with a single loss?

Explanation: Substituting this reparameterization into the Bradley-Terry preference loss produces a loss over the policy's own parameters directly β€” no separate reward model, no RL loop.

A reward model assigns scalar rewards $r_A = 2.0$ and $r_B = 0.5$ to two candidate responses. Using the Bradley-Terry formula $P(A \text{ preferred}) = \text{sigmoid}(r_A - r_B) = \dfrac{1}{1+e^{-(r_A-r_B)}}$, what is $P(A \text{ preferred over } B)$? (Round to 2 decimal places.)

Explanation: sigmoid(2.0 βˆ’ 0.5) = sigmoid(1.5) = 1/(1+e^-1.5) β‰ˆ 1/1.2231 β‰ˆ 0.82.

Using the same Bradley-Terry formula, if two responses receive equal rewards $r_A = 1.0$ and $r_B = 1.0$, what is $P(A \text{ preferred over } B)$?

Explanation: sigmoid(1.0 βˆ’ 1.0) = sigmoid(0) = 0.5 β€” equal rewards imply no preference either way.

RLHF's training objective is $\text{reward} - \beta \times \text{KL}$. If the reward model's score is 5.0, the KL divergence from the SFT policy is 0.8, and $\beta = 0.1$, what is the value of the KL-penalized objective?

Explanation: 5.0 βˆ’ (0.1 Γ— 0.8) = 5.0 βˆ’ 0.08 = 4.92.

InstructGPT's pipeline consists of how many main sequential stages (supervised fine-tuning, reward model training, and PPO-based RLHF)?

Explanation: SFT, then reward model training from preference comparisons, then PPO-based RLHF against that reward model with a KL penalty β€” three stages total.

Why is rejection sampling (best-of-$n$) fine-tuning exactly as stable as ordinary supervised fine-tuning?

Explanation: The reward model's only job is to pick which already-generated candidate to imitate; the update itself is ordinary supervised fine-tuning on self-generated, reward-filtered data, so it inherits SFT's stability rather than RL's.

What does GRPO remove from standard PPO, and what replaces it?

Explanation: Standard PPO needs a separate critic roughly the size of the policy to estimate a baseline. GRPO samples a group of completions for the same prompt and normalizes each one's reward against that group's own mean and standard deviation, eliminating the critic network entirely.

What bottleneck does RLAIF specifically address, and how?

Explanation: RLAIF has a separate language model judge which of two candidate responses is better, substituting AI-generated labels for at least part of the human-labeled preference data, achieving comparable results at a fraction of the labeling cost.