RLHF and Alignment
Teaching models what people prefer.
A helpful assistant must be more than correct. It should be clear, honest, polite, and safe. Those qualities are hard to write down as rules. So researchers found another way, learning from human preferences. In this lesson, you will meet reinforcement learning from human feedback.
- Correct is not enough
- Clear, honest, safe, helpful
- Learning from human preferences
The RLHF recipe
Here is the recipe, known as R L H F. First, the model writes several different answers to the same question. Second, human reviewers rank the answers, from best to worst. Third, a separate reward model learns to predict those rankings, so it can judge answers automatically. Fourth, the language model practises, producing answers, and receiving high scores for the ones humans would prefer. Over many rounds, its behaviour moves toward what people like.
- Generateseveral answers per question
- Rankhumans pick the best
- Reward modellearns the rankings
- Practisereinforce preferred answers
What alignment tries to achieve
This process is part of alignment, the effort to make A I systems behave in line with human values. Aligned models aim to be helpful, by really trying to solve your problem. Honest, by admitting uncertainty. And harmless, by refusing dangerous requests. It is not perfect. Models can still make mistakes, and clever users can sometimes trick them. Alignment is an active area of research.
Aligned goals
- Helpful: really solve the problem
- Honest: admit uncertainty
- Harmless: refuse dangerous requests
Remaining challenges
- Mistakes still happen
- Tricks can bypass rules
- Values differ between people
- Research is ongoing
From raw model to assistant
Let's put the whole journey together. Stage one, pre-training, builds a model that knows language and the world. Stage two, instruction tuning, teaches it to follow instructions. Stage three, learning from human feedback, polishes its behaviour, to be helpful, honest, and safe. Pre-training is by far the most expensive. The later stages are much smaller, but they make all the difference in how the model feels to use.
- Pre-traininglearns language and the world
- Instruction tuninglearns to follow instructions
- Human feedbacklearns what people prefer
- The assistant you usehelpful, honest, safer
Summary
Excellent. Reinforcement learning from human feedback ranks answers, trains a reward model, and nudges the assistant toward what people prefer. Together with pre-training and instruction tuning, it forms the full recipe behind modern assistants. Next, how these models actually produce their text, and the dial that controls creativity.
Key takeaways- RLHF: rank answers, learn a reward
- Alignment: helpful, honest, harmless
- Pre-train, instruction-tune, align