RMSNorm Scaling

Root Mean Square Normalization (RMSNorm) is a feature normalization technique that simplifies Layer Normalization by removing the mean-centering operation. It normalizes features directly using their root mean square. While mathematically simpler and computationally cheaper, RMSNorm is not completely scale-invariant if epsilon is non-zero, and it does not guarantee a zero-mean output.

Experiment

Use the RMSNorm Scaling module to observe how RMSNorm and LayerNorm behave under different input vectors and parameter settings.

1. Set the Vector control to Mixed, Epsilon to 1e-06, Gain to 1, and Coordinate to 0. Observe the calculated outputs match the worked example.
2. Change the Vector control to Offset (which adds 10 to each element of the Mixed vector). Notice how the LayerNorm output remains identical because it recenters the data, while the RMSNorm output changes significantly.
3. Change the Gain control to 2. Observe that all outputs scale proportionally.
4. Change the Vector control to Zero and Epsilon to 0. Note that both normalization techniques result in undefined behavior (division by zero).
5. Change the Vector control to Constant and keep Epsilon at 0. Observe that RMSNorm is well-defined and outputs 1, while LayerNorm remains undefined because the variance is zero.