Cost-Complexity Regression Tree Pruning
Compare every rooted subtree and see how a leaf penalty changes the selected regression tree.
Free theory articleCompare every rooted subtree and see how a leaf penalty changes the selected regression tree.
Concept & interpretation
Cost-complexity pruning is a technique used to avoid overfitting in decision trees by finding the right balance between a tree's predictive power on training data and its structural complexity. We start from a fully grown tree and prune it by removing subtrees, thereby penalizing the total number of leaves according to a parameter α.
Experiment
Follow these steps to observe the impact of the cost-complexity penalty:
- Select the Asymmetric option in the "Tree sample" dropdown control.
- Set the Penalty alpha to 0 to observe the unpruned tree with 4 leaves and SSE of 0.
- Gradually increase the Penalty alpha to 8. Observe how the optimal tree changes to the 3-leaf subtree `L|RL|RR`.
- Further increase the penalty to 50. Notice that the entire tree is pruned back to just the Root node, with 1 leaf, maximizing simplification over raw training accuracy.