Maximum Entropy Moment Matching

Finding the maximum entropy distribution over a discrete state space subject to a given expectation (moment) constraint is a foundational problem in information theory and statistical machine learning. For states {-1, 0, 1}, we seek to maximize the Shannon entropy while ensuring the expected value matches a target mean.

Exploring the Constraint with the Interactive Module

You can interact with the Maximum Entropy Moment Matching module to see how the distribution changes with the target mean.

1. Set the Target mean to 0. Notice that the Maximum Entropy distribution assigns equal probability (1/3) to all three states, resulting in the maximum possible entropy of ln(3) ≈ 1.0986.
2. Change the Distribution to Feasible two point while keeping the Target mean at 0. Notice that it assigns 0.5 to states -1 and 1, and 0 to state 0. The entropy drops to ln(2) ≈ 0.6931. The entropy deficit is about 0.4055.
3. Set the Target mean to 0.5 and the Distribution to Maximum entropy. Observe the calculated probabilities and compare them with the worked example above.