0
$\begingroup$

This question already has an answer here:

I realize that this is a relatively well published/understood concept, but I'm unfamiliar with how the conclusion is derived. Could someone ELI5 why when the data are perfectly separable, the MLE for a logistic regression $\hat{\beta} = \infty$?

$\endgroup$

marked as duplicate by SmallChess, Michael Chernick, mdewey, kjetil b halvorsen, Peter Flom - Reinstate Monica regression Apr 12 '17 at 13:26

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1
$\begingroup$

Imagine you have two data points:

  1. $x_1 = 1$, $y_1= 1$
  2. $x_2 = -1$, $y_2 = 0$

And let's say we're estimating $b$ in $P(Y= 1 \mid X = x) = \frac{e^{bx}}{1 + e^{bx}}$

How does $b=5$ do? Let $P_1$ and $P_2$ be the probability of correctly forecasting the 1st and 2nd point respectively. We have:

  • $P_2 = P_1 = \frac{e^5}{1+e^5} = .9933$

Can we do better? Sure. How about $b=6$

  • $P_2 = P_1 = \frac{e^6}{1+e^6} = .9975$

With perfect separation, you always have a higher likelihood by increasing $b$ so the maximum likelihood estimate (MLE) doesn't exist. In a sense, the MLE is $b = \infty$. See the nifty graphic on this answer: https://stats.stackexchange.com/a/224864/97925

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.