0
$\begingroup$

Logistic regression is a linear model, decision boundary generated is linear.

If the data points are linearly separable, then why does Logistic regression fail? Shouldn't it perform better on data that is actually linearly separable?

$\endgroup$
  • 3
    $\begingroup$ In what sense does logistic regression actually "fail" in perfect separation cases? Exploring that issue might help you answer this question. $\endgroup$ – whuber Jun 28 at 21:27
  • $\begingroup$ Estimating the parameters are not possible and the estimates become unstable. $\endgroup$ – Coderhhz Jun 28 at 21:29
  • 3
    $\begingroup$ But estimating the parameters is not the same problem as determining a decision boundary. Moreover, logistic regression (carefully programmed) will find a decision boundary in the case of perfect separation. These are some of the reasons you might want to think a little about why logistic regression might be applied and the specific sense in which you mean "failure." $\endgroup$ – whuber Jun 28 at 21:37
  • 1
    $\begingroup$ Logistic regression fits $E(Y|x)=P(Y=1|x)$ where the function of the linear predictor is logistic. It is not inherently a classifier, though you can make it one by drawing a line at some fitted probability (like 0.5). $\endgroup$ – Glen_b Jun 29 at 7:04
  • $\begingroup$ Logistic regression is not a classifier. It is a probability model. $\endgroup$ – Frank Harrell Jun 30 at 12:42
0
$\begingroup$

If the data are linearly separable with a positive margin, so that it can be separated by a plane in more than two (so infinitely many ways), then all those ways will maximize the probability, so the model maximizing the likelihood is not unique. So what the iterative method used to maximize the likelihood converges to is not unique. In that sense the logistic regression is unstable.

But a well implemented algorithm will find one of those solutions, and that might be good enough. And if what you want is the estimated probabilities, they will be the same for all those solutions.

So if your goal really is to find a separating plane with maximum margin, then logistic regression is the wrong method. If your goal is to estimate risk probabilities, the problem is another one: estimated risks of zero or one might be very unrealistic. So the answer to your question really depends on your goal, and you didn't tell us that.

About this question Brian Ripley in "Pattern Recognition and Neural Networks" says

We feel to much have been made of this. The difficulty is in inappropriate parameterization and the limits for infinite $|| \beta > ||$ of the fitted posterior probabilities remain perfectly suitable fits, albeit sometimes predicting probability zero or one.

See Why does logistic regression become unstable when classes are well-separated? where all this is discussed with much more detail.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.