Bilevel Ridge Hypergradient

Hyperparameter optimization can be formulated as a bilevel optimization problem. The inner problem trains the model parameters (like weights) given fixed hyperparameters (like regularization penalty). The outer problem evaluates the trained model on a validation set. By differentiating through the exact inner solution, we can compute a hypergradient to tune the hyperparameters using gradient descent.

Experimenting with the Bilevel Optimization

You can interact with the Bilevel Ridge Hypergradient module to observe how hypergradients guide penalty selection.

1. Change the Data control to "Overfit". Observe that the validation loss curve J(λ) reaches a minimum at λ = 5, which is the "Best finite lambda".
2. Adjust the Penalty lambda control to 1, then to 5, then to 10. Notice how the "Optimal train coefficient" θ*(λ) decreases as the penalty increases.
3. Change the Data control to "Opposite validation". Notice that the validation loss strictly decreases as λ grows. The optimum is an "Infimum at infinity", because the validation loss cannot reach 0, but approaches 0.5 as θ* goes to 0.
4. Set the Data control to "Aligned" and the Penalty lambda to 0. The inner model perfectly fits the validation data, yielding a validation loss of 0 and a hypergradient of 0.