Adam Moment Bias Correction
Inspect biased moving moments and the correction that changes Adam's early steps.
Free theory articleInspect biased moving moments and the correction that changes Adam's early steps.
Concept & interpretation
The Adam optimizer computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients. However, because these moving averages are initialized as zero vectors, they are biased towards zero, especially during the initial timesteps. This bias correction step addresses this issue.
Experiment
You can interact with these dynamics directly using the Adam Moment Bias Correction module.
- Set the Gradient stream to "Constant", Beta 1 to 0.9, and Beta 2 to 0.999.
- Set Updates to 1. Notice how the Uncorrected position jumps much further than the Corrected position due to the uncorrected moving averages being close to 0.
- Increase Updates to 4. Observe that the Corrected position moves linearly as intended for a constant gradient, while the Uncorrected position trace starts aggressively and gradually slows down.
- Change the Gradient stream to "Alternating". Compare how the corrected moments adapt to the changing sign.