2
$\begingroup$

I think I understand the perfect separation problem in logistic regression and answered my own question in this post from optimization perspective.

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

However, I still do not understand the p-value in such case. I saw all the values in R is <2e-16 for thousands coefficients. For example

             Estimate Std. Error    z value Pr(>|z|)
    c1       -1.524e+15  4.701e+07  -32413747   <2e-16 ***
    c2       -4.226e+15  4.735e+07  -89262659   <2e-16 ***
    c3       -2.932e+15  6.302e+07  -46524709   <2e-16 ***
    c4       -2.808e+15  4.098e+07  -68505362   <2e-16 ***
    c5        2.141e+15  7.796e+07   27470901   <2e-16 ***
    c6       -5.617e+14  7.295e+07   -7699884   <2e-16 ***
    c7       1.046e+15  7.135e+07   14654699   <2e-16 ***
    c8       1.797e+15  4.161e+07   43176668   <2e-16 ***
    c9       -1.443e+14  7.788e+07   -1852414   <2e-16 ***
    c10      2.095e+15  9.287e+07   22557866   <2e-16 ***
    c11      4.918e+14  3.600e+07   13659294   <2e-16 ***
    c12      -1.293e+14  4.204e+07   -3076600   <2e-16 ***
    ...      ...        ...         ...         ...

Why would that happen? And Can I say the p-values are not longer valid in perfect separation case?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ This isn't answerable till you explain how you're fitting the model & calculating the p-values. The common schoolboy error is using maximum-likelihood fitting together with the Wald approximation to calculate p-values in the presence of separation, typically making them absurdly large. $\endgroup$ – Scortchi - Reinstate Monica♦ Oct 25 '16 at 14:52
1
$\begingroup$

Loosely speaking your null hypothesis is that the population coefficient(s) is/are zero. Your estimate is infinite. The probability that such a value or more extreme could be drawn from a normal with mean zero and the estimated se as standard deviation is going to be much smaller than anything R is willing to print out.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, that makes sense to me. But why all coefficients go to infinity? should that be only the on the feature that can do the perfect separation? $\endgroup$ – Haitao Du Oct 25 '16 at 13:56
  • $\begingroup$ Well in act some of them are negative infinity. I think if you remove the one you know is perfect the others may look more sensible. Note however that I am not recommending removing predictors when they are perfect. $\endgroup$ – mdewey Oct 25 '16 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.