Neural Collapse Simplex Geometry
Move class clouds toward a simplex and separate within-class collapse from loss of class separation.
Free theory articleMove class clouds toward a simplex and separate within-class collapse from loss of class separation.
Concept & interpretation
During the terminal phase of training deep neural networks, the features of instances belonging to the same class often collapse to their class means, and the class means arrange themselves as the vertices of a regular simplex.
Experiment
By manipulating the parameters in the Interactive Module, you can observe how the geometry affects classification accuracy:
- Select the Center radius to 1 and the Within-class spread to 1. This eventually leads to overlap between class clusters, dropping accuracy below 1.0 (specifically 0.6666666666666666) while Collapse ratio reaches 1.0.
- Select the Center radius to 0. This causes all class means to collapse to the origin, leading to a random baseline Nearest-center accuracy of 0.3333333333333333.