Direct Preference Relative Policy

Direct Preference Optimization (DPO) optimizes a language model policy by using a reference policy and a dataset of preferences. Instead of training a separate reward model, DPO formulates the reward directly as a function of the log-ratio of the policy probabilities relative to the reference probabilities.

Experiment

You can interact with the Direct Preference Relative Policy module to observe how these values change. Follow these steps:

1. Set the Reference probability to 0.25.
2. Set the Policy probability to 0.5.
3. Set Beta to 1.
4. Set Response-one wins to 3.
5. Observe that the implicit reward difference becomes ln(3) ≈ 1.0986.
6. The predicted preference probability becomes 0.75, matching the empirical preference perfectly.
7. The policy-logit gradient becomes 0, indicating the loss is minimized.