Centered Kernel Alignment Invariance
Compare learned representations through centered sample similarities and test their invariances.
Free theory articleCompare learned representations through centered sample similarities and test their invariances.
Concept & interpretation
Let us calculate the Centered Kernel Alignment (CKA) between representations X and Y for three samples, where X = [[-1, 0], [0, 1], [1, -1]]. Under the Identity representation with translation 0 and noise 0.5 applied along the noise profile (1, -1, 0) in the first column, we add [[0.5, 0], [-0.5, 0], [0, 0]] to obtain Y = [[-0.5, 0], [-0.5, 1], [1, -1]].
Experiment
You can test different learned representations and invariances by adjusting the controls in theCentered Kernel Alignment Invariance module.
- Change the Representation to "Rotate" to apply an orthogonal transformation to the data. Notice that the Centered Kernel Alignment remains 1.0 (perfectly similar).
- Increase the Translation to shift the samples in space. CKA centers the data first, meaning it is perfectly invariant to uniform translations.
- Add Noise to break the exact linear relationship between the samples, lowering the CKA score below 1.0 as the geometric similarity decreases.
- Select the "Constant" representation, mapping all points to zero. The centered norm becomes zero, leaving CKA undefined rather than indicating perfect similarity.