Gaussian Information Bottleneck Transition
A noisy representation starts preserving useful information only after relevance becomes valuable enough to justify the compression cost.
Free theory articleA noisy representation starts preserving useful information only after relevance becomes valuable enough to justify the compression cost.
Concept & interpretation
The Gaussian Information Bottleneck defines a fundamental tradeoff for representations. A noisy encoder representation starts preserving useful information only after relevance becomes valuable enough to justify the compression cost.
Experiment
You can interact with the analytical Gaussian bottleneck tradeoff using the interactive module:Gaussian Information Bottleneck Transition.
- Set Relevance correlation to 0.5.
- Observe the Activation relevance weight threshold changes to exactly 4.
- Set Relevance weight to 2 (which is ≤ 4). Notice that theOptimal signal fraction is exactly 0. The bottleneck refuses to allocate any bit rate because the relevance return does not justify the compression cost.
- Increase Relevance weight to 8. The representation now becomes active, finding a non-zero optimal signal fraction.