Separable Logistic Finite MLE
Recognize when logistic likelihood has no finite optimizer and how noise or ridge changes existence.
Free theory articleRecognize when logistic likelihood has no finite optimizer and how noise or ridge changes existence.
Concept & interpretation
In a linearly separable classification dataset, binary outcomes can be perfectly predicted by a sufficiently large weight vector. When maximizing the logistic likelihood (or minimizing the cross-entropy loss) without regularization, the optimization attempts to push the predicted probabilities arbitrarily close to 0 or 1. This means the weights grow to infinity, and a finite Maximum Likelihood Estimator (MLE) does not exist.
Experiment
- Select Feature magnitude 1.
- Select Flip fraction 0.
- Select Ridge lambda 0.1.
- Observe the Optimum weight result is 1.6335061701558464.
The interactive module allows you to explore this phenomenon using a single-feature binary classification setup.
Adjust the Flip fraction and Ridge lambda controls to see how the objective curve transforms from having an infimum at infinity to having a well-defined minimum. You can also probe the curve at specific weights to evaluate the gradient and curvature at that point.
You can try the module here: Separable Logistic Finite MLE