EXP3 Bandit Importance Weight Feedback
Seeing only the reward you chose creates biased feedback unless rare choices receive the right importance correction.
Free theory articleSeeing only the reward you chose creates biased feedback unless rare choices receive the right importance correction.
Concept & interpretation
The EXP3 algorithm adapts full-information strategies to a bandit setting by observing only the reward for the selected action. Because it cannot see the reward of unchosen arms, it scales the observed reward using the selection probability—known as the importance weight—to maintain an unbiased estimate.
Experiment
To see this behavior in practice:
- Set the "Reward table" to "Switch" or "Ties" to observe different adversarial environments.
- Adjust the "Exploration mix" to control the minimum probability of choosing any arm.
- Change the "Seed" to observe different random decision sequences.
- Select the number of "Rounds" to see how the relative weight and probabilities update step-by-step.
- Verify the "Last observed importance estimate" and how it affects the "Arm one relative weight".