Binary Channel Mutual Information

In information theory, a noisy channel is typically modeled by the transition probabilities from an input alphabet X to an output alphabet Y. The Binary Symmetric Channel (BSC) is a common model where the input bit is flipped with probability q. The channel capacity is the supremum of the mutual information between the input and output over all possible input distributions.

Experiment

You can explore the relationship between the input bias, noise level, and mutual information in the interactive experiment.

Try the following ordered instructions to see the effects on the results:

1. Select the Flip probability dropdown and set it to 0.5 (completely noisy channel). Notice that the mutual information drops to 0, regardless of the input probability. The channel destroys all dependence between the input and output.
2. Select the Input-one probability dropdown and set it to 0 (constant input). Notice that the mutual information is 0. Since the input has no uncertainty, no information is being transmitted.
3. First set Input-one probability to 0.25, then set Flip probability to 1. The mutual information becomes approximately 0.811278124459 bits and the channel capacity is 1 bit. The channel is deterministically flipping the bits rather than destroying the information.