Bernoulli Natural Gradient
Compare Euclidean and Fisher-scaled updates without hiding invalid probabilities.
Free theory articleCompare Euclidean and Fisher-scaled updates without hiding invalid probabilities.
Concept & interpretation
When training probabilistic models, the parameter space is often better measured using information geometry rather than standard Euclidean distance. For a Bernoulli parameter p, this module compares the ordinary gradient descent update with the natural gradient step.
Experiment
Interact with the module at https://riano.app/module/bernoulli-natural-gradient to see how gradient scaling affects the probability update.
- Set Initial probability to
0.1. - Set Target probability to
1. - Set Step size to
0.5. - Observe that the ordinary raw update jumps to
5.1, moving far outside the valid probability domain. - Notice the natural gradient step keeps the new probability at a valid
0.55.
This experiment demonstrates the stability of natural gradient updates.