Covariate Shift Importance Weighting
Compute exact two-group risks, density-ratio weights and ESS dispersion, including unsupported target groups.
Free theory articleCompute exact two-group risks, density-ratio weights and ESS dispersion, including unsupported target groups.
Concept & interpretation
In machine learning, covariate shift occurs when the distribution of inputs (covariates) changes between the training (source) environment and the testing (target) environment, while the conditional distribution of outputs given inputs remains fixed. Standard empirical risk minimization on the source data can yield biased models if certain regions of the input space are underrepresented.
Experiment
You can reproduce this calculation in the Covariate Shift Importance Weighting module:
- Set Source group-one probability to
0.8. - Set Target group-one probability to
0.2. - Set Loss profile to
A. - Set Source group-one probability to
0, keeping Target group-one probability at0.2. The weighted risk is not identifiable and ESS is undefined. Set target probability to0as well: overlap holds again, the unused G=1 weight is zero, and risk is0.100000. - Restore Source group-one probability to
0.8and Target group-one probability to0.2for the numerical comparison below.
For these restored 0.8/0.2 settings, observe that the source risk is 0.74, while the oracle target risk drops to 0.26. The importance weights are w0 = 4.0 and w1 = 0.25, and the reweighted source risk matches the 0.26 target risk. Also observe the Effective Sample Size (ESS) fraction drops to 4/13 = 0.307692, an asymptotic weight-dispersion diagnostic rather than an actual finite sample count.