Nesterov Lookahead Momentum
Compare a gradient taken at a momentum lookahead with one taken at the current point.
Free theory articleCompare a gradient taken at a momentum lookahead with one taken at the current point.
Concept & interpretation
Standard momentum, often called the heavy-ball method, updates the current position using the current velocity and the gradient at the current position. Nesterov momentum improves upon this by evaluating the gradient not at the current position, but at a "lookahead" position where the momentum alone would take us.
Experiment
Explore these differences interactively in the Nesterov Lookahead Momentum module.
- Set Curvature to 1, Step size to 0.25, and Momentum to 0.5.
- Observe that at Updates 0, both Nesterov and Heavy ball methods start at position 2.
- Change Updates to 1. Both methods update to 1.5 because the initial velocity is zero, making the lookahead identical to the current position.
- Change Updates to 2. Nesterov momentum uses the gradient at the lookahead (1.25), resulting in a position of 0.9375. The Heavy ball method uses the gradient at the current position (1.5), resulting in a position of 0.875. The trajectories diverge.