Finite Neural Tangent Kernel
Measure a network's gradient kernel and test its first-order prediction of an actual training step.
Free theory articleMeasure a network's gradient kernel and test its first-order prediction of an actual training step.
Concept & interpretation
Exploring the tangent kernel and linearization error in a simple finite network.
Experiment
You can interactively explore the NTK in this small network. Open the Finite Neural Tangent Kernel module.
By selecting different "Network" presets, you can observe how different weight configurations ("Mixed", "Both active", "Dead", "Kink") impact the training dynamics.
- Change the Network preset to "Dead" and notice the gradient becomes identically zero.
- Set the Network to "Mixed", and change the Step size to 0.5.
- Observe that the error of the first-order linear approximation increases as the step size gets larger, because the actual parameters update bilinearly for even this simple ReLU network.