Prioritized Replay Importance Weights
Oversample surprising experiences and see exactly what importance weights do—and do not—correct.
Free theory articleOversample surprising experiences and see exactly what importance weights do—and do not—correct.
Concept & interpretation
Experience replay allows reinforcement learning agents to learn from past transitions. Instead of sampling uniformly, prioritized experience replay samples transitions with a higher magnitude of Temporal Difference (TD) error more frequently. To avoid bias, importance sampling weights are used to correct the expectation.
Experiment
Use the interactive module to see how importance weights affect expected updates:Prioritized Replay Importance Weights
- Set Errors to Increasing.
- Set Priority exponent to 1.
- Set Correction exponent to 1 and Normalization to Raw. Notice that the expected weighted error exactly matches the uniform target mean.
- Change Normalization to Maximum. The expected weighted error is now scaled down compared to the uniform mean.
- Change Correction exponent to 0. Notice that the expected weighted error diverges from the uniform target mean, revealing bias.