Learning Rate Schedules
Visualize how learning rate decay schedules impact optimization convergence and stability.
Free theory articleVisualize how learning rate decay schedules impact optimization convergence and stability.
Concept & interpretation
In training neural networks and other machine learning models, the learning rate controls how much to change the model in response to the estimated error each time the model weights are updated. Using a fixed learning rate often poses a dilemma: a large rate helps the model learn quickly but may cause it to overshoot the optimal solution and diverge, while a small rate ensures stable convergence but takes an impractically long time to train and risks getting stuck in local minima.
Learning rate schedules address this by systematically adjusting the learning rate over time (typically by decaying it as training progresses). This allows the model to take large steps initially for rapid exploration, and smaller steps later for fine-grained exploitation near the global minimum.