FTRL-Proximal Sparse Logistic Updates

Follow adaptive accumulators to see when an online logistic weight becomes exactly sparse. FTRL-Proximal maintains a lazy dual accumulator for gradients (z) and squared gradients (n), applying L1 regularization analytically at each step. This yields exact zero weights for features that are not sufficiently predictive, making it highly effective for massive, sparse datasets like ad click prediction.

Experiment

Use the FTRL-Proximal Sparse Logistic Updates module to see how dual accumulators create exact sparsity.

1. Set Examples to Separable, Alpha to 1, and L1 to 0.5. At 4 steps, observe that Weight one is exactly 0.0 while Weight zero is positive.
2. Change L1 to 0. Notice how Weight one is no longer zero; the sparsity is lost without L1 regularization.
3. Switch to the Sparse examples dataset. After 4 steps with L1=0, Weight zero climbs to ~0.96. The L1 penalty would have otherwise forced less predictive features to exactly zero early in training.