Your AI Knows How to Answer. But Who Teaches It What a Good Answer Is?

Hello, I'm Rijul, and I'm building LiveReview - a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. You use ChatGPT, Gemini, and all these AI bots. Whatever you ask, they usually give you a good answer, often something we like to hear. Ever wondered how they are tuned to our tastes? We humans have a hand in that too. We teach these models to produce responses that people prefer. There are different ways to teach a language model to produce responses that people prefer. Two important approaches are DPO and RLHF. Both use human preferences to guide a model, but they do it differently. Let's understand how each one works. DPO DPO stands for Direct Preference Optimization. It is a way to teach a language model which kinds of answers people prefer. When a model learns to generate text, you usually want it to produce responses that are helpful, follow instructions, and avoid undesirable behavior. But describing exactly what makes an answer "good" can be difficult. Instead, we can ask humans to compare answers. For example, given the same question, the model might produce two responses: Answer A: Clear, helpful, and directly answers the question. Answer B: Long and confusing, and doesn't really answer the question. A human can simply say: A is better than B. DPO uses many of these preferences to adjust the model so that it becomes more likely to produce responses similar to the preferred answers. How DPO works You provide training examples containing: - A prompt - A chosen answer - A rejected answer For example: Prompt: How do I reset my password? Chosen: Go to Settings → Security → Reset Password and follow the instructions. Rejected: Passwords can be changed in many different ways depending on the situation. The model learns from many such comparisons. A simple way to think about it is: "When you see situations like this, produce something more like A and less like B." After seeing enough comparisons, the model learns patterns in the preferences. The problem with DPO The model learns from the preference data it receives. If the human judgments are noisy, inconsistent, or biased, those problems can make their way into the model. For example, if humans consistently prefer overly long answers, the model may learn that longer answers are better even when they are not. So DPO is only as good as the preference data used to train it. Now let's look at another approach. RLHF RLHF stands for Reinforcement Learning from Human Feedback. Like DPO, RLHF uses human preferences to teach a model which responses people prefer. But the training process is different. A classic RLHF pipeline has three major stages. 1. Start with a base model First, you start with a pretrained language model. It already knows how to generate text, but it may not reliably follow instructions. So it is usually fine-tuned on examples of good conversations and instructions. This gives you a model that can follow instructions reasonably well. 2. Train a reward model Now humans compare different answers from the model. For example: Question: Explain photosynthesis. Answer A: Photosynthesis is the process plants use to convert light energy into chemical energy. Answer B: Plants use sunlight, water, and carbon dioxide in a biological process. Humans indicate which answer they prefer. A separate model, called a reward model, is then trained on many of these preferences. Its job is to predict how much a human would prefer a particular response. Instead of simply saying: A is better than B the reward model can assign scores to responses. For example: Answer A → 0.82 Answer B → 0.54 The reward model has learned to act as a rough approximation of the human preferences represented in its training data. 3. Practice against the reward model Now the language model generates new responses. The reward model scores those responses. The language model is then updated using reinforcement learning to produce responses that receive higher rewards. You can think of the process like this: Language model ↓ Generates an answer ↓ Reward model ↓ Gives a score ↓ Reinforcement learning ↓ Updates the language model There is also a constraint that keeps the updated model from moving too far away from its original behavior. In classic RLHF, this is commonly implemented using a KL-divergence penalty. Without such constraints, the model could find strange ways to increase its reward without actually producing better responses. A simple analogy Imagine a student learning to write essays. First, a group of teachers evaluates a collection of essays and decides which ones are better. Then, a grading assistant is trained to imitate those teachers' judgments. Now the student can write a new essay, receive a score from the grading assistant, and use that feedback to improve. The teachers don't need to grade every essay themselves. The grading assistant provides feedback repeatedly while the student practices. That is roughly the idea behind the reward-model stage of RLHF. DPO vs RLHF The biggest difference is what happens after humans provide their preferences. With RLHF, those preferences are first used to train a separate reward model. The language model then uses reinforcement learning to optimize against that reward model. With DPO, there is no separate reward-model-and-RL loop in the standard DPO procedure. The preference pairs are used directly to optimize the language model. You can think of the difference like this: RLHF Human preferences ↓ Reward model ↓ Reinforcement learning ↓ Language model DPO Human preferences ↓ Direct preference optimization ↓ Language model This makes DPO's training pipeline simpler than the classic RLHF setup. The trade-off RLHF gives you an explicit reward model and an RL optimization process, but that also makes the training pipeline more complicated. There are more moving parts, and reinforcement learning introduces its own training challenges. DPO removes the separate reward-model training and RL optimization loop, making the preference-training process simpler. But DPO still depends heavily on the quality of the preference data. In both approaches, there is a fundamental limitation: The model can only learn the preferences that are represented in the feedback it receives. If the human preferences are inconsistent, biased, or poorly defined, the resulting model can inherit those problems. Conclusion DPO and RLHF are two different ways of using human preferences to shape the behavior of a language model. RLHF uses a reward model and reinforcement learning to turn human preferences into a training signal. DPO uses preference comparisons directly to optimize the model. The important difference is not that one uses human preferences and the other doesn't. Both do. The difference is how those preferences are turned into updates to the language model. Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius - how far its impact reaches through your call graph - so you can focus attention where it actually matters. Spend code review effort where business risk is highest - not spread evenly across every diff. ⭐ Star it on GitHub: HexmosTech / LiveReview Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. blast-radius-demo.mp4 LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. Here's the goal: - A 3-line fix in a function used by 40 other files, that also writes to a database, should score high. - A 300-line UI change in one file, fully covered by… Click below to try LiveReview with your codebase: Top comments (1) The reward model as 'grading assistant' analogy is a good one for making the RLHF pipeline concrete without losing the key insight: the teachers don't grade every essay, but the grading assistant is only as good as the subset of essays the teachers evaluated. That limitation carries through to the language model. The core distinction you're drawing between RLHF and DPO is right: RLHF adds an intermediate reward model that approximates human preferences, which then provides the training signal for RL optimization. DPO cuts that intermediate step and uses preference pairs directly. Simpler pipeline, but the same fundamental constraint: the model inherits whatever biases and inconsistencies are in the preference data. The point about length bias is worth highlighting. If human raters consistently prefer longer answers, the model learns longer is better - even when conciseness would serve the user better. This is one of the known failure modes in preference-trained models, and it's hard to detect in the aggregate because the model is doing exactly what the training signal asked for. The KL-divergence penalty in RLHF is doing real work that's easy to miss in an intro explanation. Without it, reinforcement learning can find reward-hacking strategies - technically maximizing the reward model's score while producing outputs that no human would prefer. The constraint that keeps the model close to its original behavior is what makes the RL loop usable in practice.