2
$\begingroup$

I think I understand the perfect separation problem in logistic regression and answered my own question in this post from optimization perspective.

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

However, I still do not understand the p-value in such case. I saw all the values in R is <2e-16 for thousands coefficients. For example

             Estimate Std. Error    z value Pr(>|z|)
    c1       -1.524e+15  4.701e+07  -32413747   <2e-16 ***
    c2       -4.226e+15  4.735e+07  -89262659   <2e-16 ***
    c3       -2.932e+15  6.302e+07  -46524709   <2e-16 ***
    c4       -2.808e+15  4.098e+07  -68505362   <2e-16 ***
    c5        2.141e+15  7.796e+07   27470901   <2e-16 ***
    c6       -5.617e+14  7.295e+07   -7699884   <2e-16 ***
    c7       1.046e+15  7.135e+07   14654699   <2e-16 ***
    c8       1.797e+15  4.161e+07   43176668   <2e-16 ***
    c9       -1.443e+14  7.788e+07   -1852414   <2e-16 ***
    c10      2.095e+15  9.287e+07   22557866   <2e-16 ***
    c11      4.918e+14  3.600e+07   13659294   <2e-16 ***
    c12      -1.293e+14  4.204e+07   -3076600   <2e-16 ***
    ...      ...        ...         ...         ...

Why would that happen? And Can I say the p-values are not longer valid in perfect separation case?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ This isn't answerable till you explain how you're fitting the model & calculating the p-values. The common schoolboy error is using maximum-likelihood fitting together with the Wald approximation to calculate p-values in the presence of separation, typically making them absurdly large. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Oct 25, 2016 at 14:52

1 Answer 1

Reset to default
1
$\begingroup$

Loosely speaking your null hypothesis is that the population coefficient(s) is/are zero. Your estimate is infinite. The probability that such a value or more extreme could be drawn from a normal with mean zero and the estimated se as standard deviation is going to be much smaller than anything R is willing to print out.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, that makes sense to me. But why all coefficients go to infinity? should that be only the on the feature that can do the perfect separation? $\endgroup$
    – Haitao Du
    Oct 25, 2016 at 13:56
  • $\begingroup$ Well in act some of them are negative infinity. I think if you remove the one you know is perfect the others may look more sensible. Note however that I am not recommending removing predictors when they are perfect. $\endgroup$
    – mdewey
    Oct 25, 2016 at 14:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.