Adaptive Decoupled Weight Decay
With an adaptive metric, L2 regularization and multiplying weights by a decay factor become different updates.
Free theory articleWith an adaptive metric, L2 regularization and multiplying weights by a decay factor become different updates.
Concept & interpretation
In modern deep learning optimization, weight decay is often applied to regularize the model. When using standard stochastic gradient descent, L2 regularization (adding an L2 penalty to the loss) and weight decay (subtracting a fraction of the weight at each step) are equivalent. However, when an adaptive preconditioner like Adam is introduced, these two approaches lead to geometrically distinct updates, motivating the development of decoupled weight decay (e.g., AdamW).
Experiment
Use the interactive controls to see how updates change under different configurations.
- Select Second preconditioner to 0.1, Step size to 0.1, and Decay strength to 1.
- Observe that the Coupled second coordinate is 1.91, while the Decoupled second coordinate is 1.73, making the difference -0.18.
- Change the Second preconditioner to 1. You will see both updates agree with second coordinates of 1.1, showing that isotropic preconditioners eliminate this difference.