ReLU Sharpness Rescaling Symmetry
Keep a neural function identical while making its parameter-space minimum look sharper.
Free theory articleKeep a neural function identical while making its parameter-space minimum look sharper.
Concept & interpretation
This module explores a mathematical quirk in neural networks: changing the scale of parameters can keep the exact same function behavior while radically altering how "sharp" or "flat" the training landscape appears around that solution.
Experiment
To see this symmetry in action, open the module at https://riano.app/module/relu-sharpness-rescaling-symmetry.
Follow these steps:
- Change the Rescaling control to adjust c. Observe that the Original Prediction always matches the probe value, meaning the network function is unchanged.
- Set a small Perturbation (like 0.1) and watch how the First-weight perturbation loss drastically changes depending on the rescaling factor, even though the unperturbed model is identical.
- Change the Probe input to verify the model acts as the identity for positive values and zero for negative values.