Nathan Lambert’s Reinforcement Learning from Human Feedback is the clearest guide yet to how chatbots are turned from text predictors into products with a personality. A new study that traces sycophancy through Lambert’s own open models tests the book’s map of where that personality comes from. It also shows why the book is still the right place to start.
Ask almost anyone why chatbots flatter their users and you will hear the same answer: reinforcement learning from human feedback (RLHF). People rate answers, people like being agreed with, and the model learns to agree. That account gained wide currency after OpenAI rolled back a GPT-4o update in April 2025. The company said the update had added a reward signal based on ChatGPT users’ thumbs-up and thumbs-down ratings, and that “user feedback in particular can sometimes favor more agreeable responses, likely amplifying the shift we saw.”
A preprint posted to arXiv on 5 November 2026 argues that the standard story is mostly wrong. In “Don’t Blame The Preferences”, Chao Péter Yang, Catherine Zehr, Pranav Bijith and Brandon Fain of Duke and Yale measured sycophancy at every released training checkpoint of three fully open models. They conclude that “preference alignment is likely not the primary cause of model sycophancy”. By their measurements the behavior is largely in place before preference tuning begins, after pretraining and supervised instruction tuning.
Two of the three model families they examined, Tülu 3 and OLMo 3, come from the Allen Institute for AI (Ai2), where Nathan Lambert built open post-training recipes. Lambert was the lead author of the Tülu 3 report. His book, published in print by Manning in July 2026 and still free to read at rlhfbook.com, is the most complete public account of how such recipes are made. So the timing makes a useful test: does the book give readers what they need to make sense of a finding like this? For this review I read the current web edition (last built on 5 November 2026). I read the introduction, the over-optimization and model-character chapters and Appendix B in full, and the relevant sections of the chapters on instruction tuning, preference data, synthetic data, regularization and evaluation and of the practical-issues appendix. I sampled the algorithm chapters. Mostly, the answer is yes, though not always in the chapters where you would look first.
What the book is
The Manning edition runs to 312 pages over 17 short chapters, with a foreword by Hugging Face co-founder Thomas Wolf. Its web version is licensed under Creative Commons (CC BY-NC-SA 4.0), and its source is on GitHub. It comes with a codebase and a recorded lecture course. The title is narrower than the contents. Lambert says on the book’s homepage that he is considering a second edition that would “rebrand it as a post-training book, rather than a RLHF book.” That is a fair description of what he has already written.
The structure follows the canonical recipe that OpenAI set out with InstructGPT in 2022. First comes instruction fine-tuning, which teaches a raw pretrained model to answer in a question-and-answer format. Then a reward model is trained on human judgments of which of two answers is better. Finally, reinforcement learning pushes the model toward answers that the reward model scores highly. Around that spine Lambert adds chapters on direct alignment algorithms such as DPO (which skips the separate reward model), rejection sampling, reasoning models trained with reinforcement learning with verifiable rewards (RLVR, a term his Tülu 3 team coined), tool use, synthetic data and distillation, evaluation, and what he calls character training.
He is clear about the book’s limits. Chapter 1 calls it “not a comprehensive textbook, but rather a quick book for reminders and getting started,” written for readers with “entry level experience with language modeling, reinforcement learning, and general machine learning.” In practice the bar is higher than that suggests. The reinforcement learning chapter alone accounts for about 16,600 of the roughly 85,000 words in the web edition’s source files, and it moves from the policy-gradient derivation through PPO, GRPO, GSPO and CISPO with code. A reader without calculus and some PyTorch will skim it.
The central argument: post-training is not just style
Underneath the tutorials, the book makes an argument. In 2023 a common view held that alignment is cosmetic. The LIMA paper’s “Superficial Alignment Hypothesis” proposed that a model’s knowledge comes almost entirely from pretraining and that alignment just selects a format. Lambert takes this on directly in Chapter 1: “The superficial alignment hypothesis is wrong for the same reason that people who think RLHF and post-training are just for vibes are still wrong.”
His evidence is mostly from compute and results. He notes that DeepSeek R1 spent about 147,000 H800 GPU hours on reinforcement learning, roughly 5% of the 2.8 million GPU hours used to pretrain its base model. By 2026, he writes, single ablation experiments in RL research can take 10,000 to 100,000 GPU hours. He also cites Ai2’s own experience: OLMoE’s benchmark average went from 35 to 48 after a post-training update alone. Appendix B, “Beyond ‘Just Style’”, adds that style is part of what information is, not decoration.
The argument is persuasive, and it is the book’s main contribution to public debate. If post-training shapes behavior and not just tone, then the choices labs make at that stage are consequential choices about products, not technical housekeeping.
Where it is strongest: failure modes and incentives
The second half is the most valuable part for readers outside the labs. Chapter 14, on over-optimization, explains Goodhart’s law in its RLHF form. A reward model is only a proxy for what users want, and reinforcement learning is a strong enough optimizer to exploit the gap. Lambert does not soften the conclusion: “Over-optimization is fundamental and unavoidable with RLHF due to the soft nature of the reward signal.” He lists the familiar symptoms (“Certainly!”, hedging, over-apologizing, sycophancy, over-refusal) and uses the GPT-4o episode as a case study of misalignment.
Chapter 16, on evaluation, should be read by anyone who reports on model launches. It explains why benchmark tables from different labs cannot be compared directly. Prompts, sampling settings, system prompts and inference budgets differ, and labs do not say which test sets they trained against. Citing Ai2’s Olmo 3 work, Lambert notes that most post-training evaluations vary by 0.25 to 1.5 points even with the setup held fixed. His summary is blunt: “Language model evaluation stacks are perceived as marketing because the evaluations have no hard source of truth.”
Chapter 17, on model character, is where the book is most candid about commercial incentives. Lambert describes how Anthropic and others train a stable persona into models through synthetic data pipelines related to Constitutional AI. Then he draws the conclusion that a lab would not: “From an industry perspective, it seems more likely that RLHF generally is about capturing the upside of methods like character training as a performance tool for capturing users’ interests, rather than a safety one.” He adds that “the methods used for character training can instill any trait into models, not just positive ones,” and he says the field does not yet know what character training does to models, something we need to know “in order to know how AI companies change the models to maximize engagement and other user-facing metrics.” For a book written by someone who builds these systems, that is an unusually direct statement about engagement incentives.
Where the new sycophancy study tests it
This is where the Duke–Yale study comes in. The authors ran three published sycophancy benchmarks on the base, instruction-tuned (SFT), preference-tuned and final checkpoints of SmolLM3 3B, OLMo 3 7B and Tülu 3 8B. The benchmarks cover backing down on correct answers under pushback, agreeing with users’ stated opinions, and “social sycophancy”, meaning excessive validation in personal advice. Across all nine model-and-benchmark combinations, the authors report that the SFT checkpoints already show “at least 92% of the preference-aligned model’s sycophancy score.” Preference tuning raised sycophancy in only four of the nine cases, by 4.2 percentage points on average when it did.
The jumps at SFT are large. On one validation measure for open-ended advice questions, OLMo 3 went from 26.4% at the base checkpoint to 83.9% after SFT, against a human baseline of 19.3% on the same rubric. The authors also used an interpretability method to score training datasets against a “sycophancy direction” inside each model. They found that SFT data showed a similar per-example level to preference data but made up 7 to 43 times as many training examples. In all three families, the four highest-scoring datasets were SFT datasets. In OLMo 3’s case, the largest single increase on the opinion-agreement benchmark, 11.7 points, came from the final RLVR stage, which trains against automatically checkable rewards rather than human preferences. None of the simple accounts predicts that.
Read against this, the book’s framing shows a gap. Where Lambert discusses sycophancy, it sits in the preference chapters. Chapter 11 treats it as a bias in human labels, one that reflects “issues within humans that are often outside of the labeling criteria.” Chapter 14 says “agree-with-the-user” behaviors “can be reinforced by preference data that overweights being supportive or confident relative to being accurate.” That wording is hedged, and none of it is contradicted by the study. But the book’s basic division of labor points readers toward preference tuning. Chapter 1 says instruction tuning provides the question-answer format, and that “RLHF is what takes these answers and crafts them into the reliable, warm, and engaging answers we now expect.” Chapter 4, on instruction tuning, is almost entirely about chat templates, loss masking and hyperparameters. It does not discuss what behavioral traits an SFT dataset might carry.
The book still has the material for a better explanation, but it is in Chapter 12. There Lambert writes that for instruction data, “synthetic generation has largely won — distillation from stronger models now produces higher quality completions than most human writers can provide at scale.” The SFT datasets in open recipes like Tülu 3 are largely generated by frontier models, which have themselves been through preference tuning and character training. The study’s authors flag the same issue as a limitation: “a sizable portion of the SFT datasets used in these training recipes are synthetically generated by frontier models. Those models have themselves undergone preference alignment.”
Combine Chapter 12 with Chapter 17 and a reader can see the likely mechanism. Sycophancy may not be learned fresh at each lab’s preference stage. It may be passed down, inherited through distilled instruction data from teachers whose personalities were tuned for user approval. Neither the book nor the paper proves this. The paper cannot see inside the frontier models, and its evidence is limited to three small open models judged partly by other language models. But it changes where accountability sits. The study argues the point itself: if SFT is the main channel, developers’ data-curation choices matter more than raters’ preferences. The book’s chapters support that reading, but the book never puts them together.
What it gets wrong, or leaves out
Apart from the sycophancy framing, the weaknesses come with the format. The book moves fast, and its 2026 material sits on top of a structure built in 2024. The changelog shows a major reorganization in January 2026, a direct-alignment chapter in February, and late additions on multi-teacher on-policy distillation and agentic post-training that the web version has but the first print run may not. Print readers get a snapshot of a moving field. Lambert’s own errata list shows how much has changed since July.
Some claims also rest on reputation more than evidence. Chapter 1 says Anthropic “now has arguably the best post-training” and that the companies that adopted RLHF early “ended up winning out.” Those are reasonable insider judgments, but they are not demonstrated in the text. The book also tells its history largely through Ai2 and Hugging Face projects that Lambert worked on, such as Zephyr, Tülu and Olmo. That is a strength when he explains decisions he saw being made. It narrows the view when the subject is what closed labs actually do, which by his own account is mostly undisclosed.
Finally, the book offers little on governance. Chapter 17 calls model specifications “one of the few tools” for comparing a model’s behavior with what its designers intended. It notes that “a major unknown with model specs and related documents is the effort that model developers put into making the model follow them.” That is the right question, but the book does not pursue it. Readers looking for how sycophancy, persona drift or engagement optimization should be audited from outside a lab will find the vocabulary here, not the answers.
Who should read it
Engineers starting post-training work should read it front to back. It is the best single reference for how the stages fit together, and the free web edition with its code makes it easy to use as a working reference. Researchers will mostly use it as an annotated bibliography with strong opinions. Product managers, policy staff and journalists can skip most of Chapters 5 to 9 and read Chapters 1, 11, 12, 14, 16 and 17 and Appendices B and C. Those chapters explain why chatbots behave as they do, why benchmark claims deserve skepticism, and why, as Appendix C notes, developing a recipe can cost “10X to 100X the compute of the final few training runs.”
The sycophancy study shows what the book is for. It does not settle where a chatbot’s flattery comes from, and it does not claim to. But a reader who has worked through it can read the Duke–Yale paper and recognize three things. A result from three open models is an important clue, not a verdict. Fully open training recipes like Lambert’s are the only reason the question can be asked at all. And the answer probably runs through the synthetic data that models now learn from one another. That is a good outcome for a book that started as a website. The next edition should move its account of model personality out of the preference chapters and into the data pipeline, where Chapter 12 already places most of the evidence.
Book details
Reinforcement Learning from Human Feedback: LLM alignment and post-training, by Nathan Lambert, foreword by Thomas Wolf. Manning Publications, July 2026. 312 pages. ISBN 9781633434301. Free web edition at rlhfbook.com.