Direct Preference Optimization: Your Language Model is Secretly a Reward Model —

DPO introduces a simple classification loss that directly optimizes language model policies on human preference data, eliminating the need for reinforcement learning while maintaining theoretical equivalence to the RLHF objective.