Variational KL and Exponential Tilting

The Kullback-Leibler (KL) divergence, KL(P||Q), measures the difference between two probability distributions P and Q. By applying Gibbs' variational inequality, we can turn any arbitrary score function into a lower bound on the KL divergence, and explain the gap through the exponentially tilted law.

Experiment

You can reproduce this behavior using the interactive module controls. Open the module at https://riano.app/module/variational-kl-exponential-tilt and follow these steps:

1. Set the Laws control to "Crossed".
2. Set the Score control to "Log density ratio".
3. Observe that the Bound gap drops to exactly 0.0. The variational bound is tight (equal to the True KL) when the chosen score is precisely the log density ratio log(P/Q).
4. Now change the Scale control to "2" and note that the bound becomes looser (Bound gap > 0).
5. Adjust the Offset control (e.g., to "1" or "-1"). Notice that the Log partition changes, but the Variational bound, True KL, Bound gap, and the tilted probabilities (R) remain completely unaffected by additive constants.