PAC–Bayes KL Risk Certificate
Balance empirical errors and posterior complexity with a finite-sample Gibbs-risk certificate.
Free theory articleBalance empirical errors and posterior complexity with a finite-sample Gibbs-risk certificate.
Concept & interpretation
The PAC-Bayes KL theorem bounds the true generalization error (the risk) of a randomized hypothesis, known as the Gibbs classifier, given its empirical performance and a measure of its complexity compared to a prior expectation.
Experiment
You can verify how changing the complexity of the posterior distribution and the size of the training dataset affects the risk budget and upper risk bound using the interactive PAC–Bayes KL Risk Certificate module.
- Select the Posterior control and change it from "Good" to "Point" (which concentrates all mass on a single hypothesis).
- Observe that the Posterior prior KL increases from 0.459 to 1.098, reflecting a higher complexity penalty for drastically shifting away from the uniform prior.
- Change the Training errors to "Wrong". The Empirical Gibbs loss instantly jumps to 1, and consequently, the PAC Bayes upper risk hits the ceiling at 1.
- Change the Sample size from 100 to 200. Observe the Bernoulli KL budget decrease as more data reduces the uncertainty, tightening the upper risk bound closer to the empirical loss.