Logistic regression is a linear model, decision boundary generated is linear.
If the data points are linearly separable, then why does Logistic regression fail? Shouldn't it perform better on data that is actually linearly separable?
$\begingroup$
$\endgroup$
5
-
5$\begingroup$ In what sense does logistic regression actually "fail" in perfect separation cases? Exploring that issue might help you answer this question. $\endgroup$– whuber ♦Jun 28, 2019 at 21:27
-
$\begingroup$ Estimating the parameters are not possible and the estimates become unstable. $\endgroup$– CoderhhzJun 28, 2019 at 21:29
-
4$\begingroup$ But estimating the parameters is not the same problem as determining a decision boundary. Moreover, logistic regression (carefully programmed) will find a decision boundary in the case of perfect separation. These are some of the reasons you might want to think a little about why logistic regression might be applied and the specific sense in which you mean "failure." $\endgroup$– whuber ♦Jun 28, 2019 at 21:37
-
1$\begingroup$ Logistic regression fits $E(Y|x)=P(Y=1|x)$ where the function of the linear predictor is logistic. It is not inherently a classifier, though you can make it one by drawing a line at some fitted probability (like 0.5). $\endgroup$– Glen_bJun 29, 2019 at 7:04
-
3$\begingroup$ Logistic regression is not a classifier. It is a probability model. $\endgroup$– Frank HarrellJun 30, 2019 at 12:42
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If the data are linearly separable with a positive margin, so that it can be separated by a plane in more than two (so infinitely many ways), then all those ways will maximize the probability, so the model maximizing the likelihood is not unique. So what the iterative method used to maximize the likelihood converges to is not unique. In that sense the logistic regression is unstable.
But a well implemented algorithm will find one of those solutions, and that might be good enough. And if what you want is the estimated probabilities, they will be the same for all those solutions.
So if your goal really is to find a separating plane with maximum margin, then logistic regression is the wrong method. If your goal is to estimate risk probabilities, the problem is another one: estimated risks of zero or one might be very unrealistic. So the answer to your question really depends on your goal, and you didn't tell us that.
About this question Brian Ripley in "Pattern Recognition and Neural Networks" says
We feel to much have been made of this. The difficulty is in inappropriate parameterization and the limits for infinite $\|\beta\|$ of the fitted posterior probabilities remain perfectly suitable fits, albeit sometimes predicting probability zero or one.
See Why does logistic regression become unstable when classes are well-separated? where all this is discussed with much more detail.
answered Jun 30, 2019 at 11:31