There are already posts about warnings from R dealing with logistic regression and linear separation such as this one. I just wanna make sure if in Python Scikit Learn this problem is all solved by the L1/L2 regularization part in the optimization function. In other words, it's safe to say users will not get any infinite MLE estimates warnings from sklearn.linear_model.LogisticRegression?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Yes, sklearn.linear_model.LogisticRegression uses penalized logistic regression which "solves" the problem of perfect separation. If you set C to something too large, you might still end up with bad results, though.
-
$\begingroup$ Why we need to bother with regularization when data are linearly separable? I mean the weights can go to infinity but the decision boundary won't change. $\endgroup$ Commented Dec 19, 2022 at 13:23
-
1$\begingroup$ @adosar Two things here: computational and statistical. Computationally (what this question was about), it's annoying to have your weights go to infinity; small regularization prevents that. Statistically, remember that if the dimension is greater than the number of data points, then almost any dataset (in the measure theoretic sense) linearly separable; this is an example where you might expect regularization to be important for finding the "right" decision boundary to generalize well. (If the dimension is high but less than $n$, it can still be very helpful, depending.) $\endgroup$– DanicaCommented Dec 31, 2022 at 9:05