Doubly Robust Policy Value
Combine a reward model with importance-weighted residuals to estimate a target policy value.
Free theory articleCombine a reward model with importance-weighted residuals to estimate a target policy value.
Concept & interpretation
Policy evaluation in reinforcement learning and contextual bandits estimates the value of a target policy using logs collected from a different behavior policy. The doubly robust estimator combines two models: an inverse probability weighting (IPW) on the logging distribution and a direct reward regression (Q-hat).
Experiment with Estimators
You can test different nuisance models in the interactive Doubly Robust Policy Value module.
- Set Target action-one probability to 0.25.
- Set Behavior estimate to "Correct" and Reward estimate to "Constant half". Observe that the Doubly Robust value is exactly 1.375.
- Change Behavior estimate to "Low action one". Because the reward estimate is wrong, the IPW estimator becomes biased (2.583). Since the reward model is also wrong, the DR estimator may be biased, but happens to remain correct in this particular deterministic fixture.
- Change Reward estimate to "Zero". Now both nuisance models are incorrect, and the DR estimator yields a biased value of 2.583.