Elastic Weight Consolidation Toy
Balance a new quadratic task against parameter importance from an old task.
Free theory articleBalance a new quadratic task against parameter importance from an old task.
Concept & interpretation
Elastic Weight Consolidation (EWC) provides a mechanism to train neural networks on sequential tasks without catastrophically forgetting what was learned in earlier tasks. By retaining parameter importance across tasks, models balance acquiring new knowledge with preserving existing capabilities.
Experiment
You can interactively explore the effect of λ on the parameter optimization trajectory in the Elastic Weight Consolidation Toy module.
- Set Tasks to "Complementary".
- Set Retention strength (λ) to "4".
- Observe how the consolidated minimizer shifts closer to the old task minimizer compared to λ = 1. The old task loss L_A decreases to 0.972.
- Set Tasks to "Unidentified coordinate". Notice how if a parameter has zero importance in both tasks (when λ = 0), the minimizer is technically non-unique.