Target Encoding and Leakage
Track exactly which targets are allowed into a categorical encoding.
Free theory articleTrack exactly which targets are allowed into a categorical encoding.
Concept & interpretation
Target encoding (or mean encoding) is a powerful technique to handle categorical features with high cardinality. Instead of creating many one-hot columns, each category is replaced with the mean of the target variable for that category. However, calculating this mean on the same dataset used for evaluation can lead to severe data leakage.
Experiment
Explore the impact of the protocol in the interactive Target Encoding and Leakage module.
- Set the Dataset to "Singletons" and Row index to 0.
- Choose "In sample" as the Protocol. Note the Encoded value is 0.0, which perfectly matches the Observed target, resulting in 0 error. This is leakage.
- Change the Protocol to "Leave one out". The Encoded value changes to 0.6, utilizing the Training prior, as the group count becomes 0. The error is now 0.36.
- Adjust the Smoothing parameter to see how alpha values pull the encoded value closer to the global prior when group counts are small.