Kullback Leibler Divergence

Suppose we have two unnormalized integer weight vectors over 3 categories:
P weights = [2, 1, 1]
Q weights = [1, 2, 1]

Interactive Experiment

Explore the interactive KL Divergence module here: KL Divergence Visualizer.

In this module, you can modify the unnormalized weights of two 3-category distributions P and Q.

1. Observe Asymmetry: Set P weight 1 to 2, P weight 2 to 2, P weight 3 to 0. Then set Q weight 1 to 1, Q weight 2 to 3, Q weight 3 to 0. You will see that D(P||Q) is ~0.2075 bits, while D(Q||P) is ~0.1887 bits. This clearly demonstrates the asymmetry of KL Divergence.
2. Zero Probabilities: Set P weight 1 to 1, P weight 2 to 0, P weight 3 to 3. Then set Q weight 1 to 1, Q weight 2 to 1, Q weight 3 to 1. D(P||Q) is finite. However, D(Q||P) becomes Positive Infinity because Q has non-zero probability for category 2, while P has zero probability there. A missing reference support creates infinity.
3. Identical Distributions: Set P and Q to have the same weights. Notice that both divergences drop to 0 bits.