AdaGrad Coordinate Learning Rates
Watch accumulated gradients adapt each coordinate’s step size.
Free theory articleWatch accumulated gradients adapt each coordinate’s step size.
Concept & interpretation
Adaptive Gradient Algorithm (AdaGrad) dynamically adjusts the learning rate for each parameter by keeping a running sum of the squared gradients.
experiment
In the interactive AdaGrad Coordinate Learning Rates module, you can observe this behavior:
- Set the Y curvature to 4 and the Learning rate to 0.5.
- Observe the trajectory for different step counts (1, 2, 5).
- Notice how the effective learning rates decrease over time, which can eventually lead to very slow progress.