Gumbel–Softmax Relaxation
Move a relaxed categorical draw through a simplex as temperature changes.
Free theory articleMove a relaxed categorical draw through a simplex as temperature changes.
Concept & interpretation
The Gumbel-Softmax trick provides a continuous, differentiable approximation to a discrete categorical distribution. By relaxing a discrete draw into a continuous sample on a probability simplex, we can backpropagate through stochastic nodes in neural networks.
Experiment
Observe the effect of the temperature parameter in the interactive Gumbel-Softmax module.
- Start with Logits set to Favor A, Noise set to Tilted draw, and Temperature set to 1. The relaxed probabilities spread across the classes, but strongly favor A. The expected numerical result Relaxed A is 0.9055519139212392.
- Change the temperature to 0.25. Notice how the point on the simplex sharply converges towards the corner representing class A. The entropy of the relaxed probabilities drops significantly.
- If you set logits to "Equal" and noise to "Equal draw", all classes tie. As temperature decreases, the relaxed output stays uniformly distributed because of the exact tie, ensuring we do not falsely promise a one-hot limit when an argmax is ambiguous.