Label Smoothing and Cross-Entropy

Label smoothing is a regularization technique that prevents a model from becoming overconfident by mixing the true label with a uniform distribution.

Experiment

Explore the effects of varying epsilon on predictions and gradients in the interactive module:

1. Change the Logit 0 value and observe how the probabilities change. Notice how the gradient adjusts based on the discrepancy between the target q and the predicted probability p.
2. Change the True class to 1 and observe how the smoothed targets update, shifting the maximum target mass.
3. Set Smoothing quarters to 4 (which means ε = 1). Observe that the targets become completely uniform ([1/3, 1/3, 1/3]), independent of the true class.
4. Set Smoothing quarters to 0 (which means ε = 0). Observe that the targets recover the exact one-hot vector and the cross-entropy loss equals the hard-label loss.