0
$\begingroup$

I am running a logistic regression, but for a group of predictors I tried, all of then are highly significant but the residual deviance is much higher than the null deviance.

glm(formula = label ~ pitch_0 + pitch_1 + pitch_2 + pitch_3 + 
pitch_4 + pitch_5 + pitch_6 + pitch_7 + pitch_8 + pitch_9 + 
pitch_10 + pitch_11 + pitch_12 + pitch_13 + pitch_14 + pitch_15 + 
pitch_16 + pitch_17 + pitch_18 + pitch_19, family = "binomial", 
data = sh_missLDA_wLabel)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
  -8.49   -8.49    0.00    0.00    8.49  

Coefficients:
          Estimate Std. Error   z value Pr(>|z|)    
(Intercept)  4.276e+14  5.339e+06  80083623   <2e-16 ***
pitch_0      1.435e+12  2.615e+04  54860285   <2e-16 ***
pitch_1     -5.965e+11  2.735e+04 -21807013   <2e-16 ***
pitch_2      2.007e+11  3.127e+04   6418959   <2e-16 ***
pitch_3      1.539e+12  3.187e+04  48278273   <2e-16 ***
pitch_4      3.260e+12  3.197e+04 101963670   <2e-16 ***
pitch_5     -2.749e+12  3.458e+04 -79518779   <2e-16 ***
pitch_6     -2.814e+12  3.831e+04 -73455897   <2e-16 ***
pitch_7      2.437e+12  3.950e+04  61683535   <2e-16 ***
pitch_8     -8.490e+11  4.124e+04 -20586227   <2e-16 ***
pitch_9     -5.832e+11  4.184e+04 -13938449   <2e-16 ***
pitch_10    -4.731e+11  4.403e+04 -10745271   <2e-16 ***
pitch_11    -5.542e+11  4.564e+04 -12142293   <2e-16 ***
pitch_12     2.566e+12  4.609e+04  55679463   <2e-16 ***
pitch_13    -5.286e+11  4.688e+04 -11273537   <2e-16 ***
pitch_14     6.256e+11  4.704e+04  13299313   <2e-16 ***
pitch_15    -9.796e+11  4.725e+04 -20730533   <2e-16 ***
pitch_16    -1.303e+12  4.770e+04 -27309726   <2e-16 ***
pitch_17     3.202e+11  4.806e+04   6661272   <2e-16 ***
pitch_18    -2.219e+12  4.888e+04 -45390341   <2e-16 ***
pitch_19    -1.663e+12  4.928e+04 -33751383   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance:  218.93  on 157  degrees of freedom
Residual deviance: 5334.46  on 137  degrees of freedom
AIC: 5376.5

Number of Fisher Scoring iterations: 11

I am wondering in this case, what does this result implying? From my previous google search, it seems to say that if this indicates a lack of fit to the data but why is the case that all of the predictors are significant to with p-value < 2e-16

More details: the pitch variables are the first 25 components from PCA, where pitch variables are initially in very high dimensions of more than 10,000 per data point and the first 25 components correspond to more than 50% of the variation.

I also tried to put in pitch variable at a time and see the results, only pitch_19 and pitch_20 are significant, reported below.

glm(formula = label ~ pitch_20, family = "binomial", data = 
sh_missLDA_wLabel)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.990  -1.195   0.790   1.154   1.458  

Coefficients:
             Estimate Std. Error z value Pr(>|z|)  
 (Intercept)  0.014154   0.163925   0.086    0.931  
  pitch_20    -0.006809   0.003519  -1.935    0.053 .
 ---
 Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

 (Dispersion parameter for binomial family taken to be 1)

  Null deviance: 218.93  on 157  degrees of freedom
  Residual deviance: 213.02  on 156  degrees of freedom
  AIC: 217.02

  Number of Fisher Scoring iterations: 5

and

  glm(formula = label ~ pitch_19, family = "binomial", data = 
  sh_missLDA_wLabel)

  Deviance Residuals: 
       Min       1Q   Median       3Q      Max  
    -1.5029  -1.2041   0.8899   1.1443   1.5430  

  Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)  0.020015   0.163907   0.122   0.9028  
 pitch_19    -0.005456   0.003279  -1.664   0.0961 .
 ---
 Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

 (Dispersion parameter for binomial family taken to be 1)

 Null deviance: 218.93  on 157  degrees of freedom
 Residual deviance: 214.44  on 156  degrees of freedom
 AIC: 218.44

 Number of Fisher Scoring iterations: 5
$\endgroup$
1
$\begingroup$

Your coefficients are all enormous in magnitude suggesting that in fact they are tending to infinity or minus infinity. This in turn suggests that the problem is separation. For some combination of your predictors you have perfect prediction. There are many posts on this site tagged which offer guidance. Of course if that does not help then edit your question with more details to explain what else might be happening.

Edit in response to comments by the OP

One way of exploring the issue further would be to refit the model deleting each variable in turn. So if there are currently 20 variables you would end up with 20 new models based on 19 variables. Then examine these. If the separation is due to one variable then the model without that variable will now look very different. If it is due to the linear combination of several variables then it is possible that all the the models involving those variables will now look OK. Then at least you know which variables to investigate further. Note that this is not the same as the automatic application of step-wise methods.

$\endgroup$
  • $\begingroup$ are there ways to know which combinations of predictors lead to perfect separation since I did not get the warning message on: glm.fit: fitted probabilities numerically 0 or 1 occurred $\endgroup$ – lll Sep 16 '18 at 21:30
  • $\begingroup$ I think I would have tried it the other way. Start with all 20 of them and then delete each in turn. $\endgroup$ – mdewey Sep 17 '18 at 16:26
  • $\begingroup$ I know I can use AIC to select the variables step-wise, but is that the step-wise selection in general not a very method based on a question I asked earlier? $\endgroup$ – lll Sep 17 '18 at 22:37
  • $\begingroup$ I have edited my answer to provide extra suggestions. $\endgroup$ – mdewey Sep 18 '18 at 12:42
  • $\begingroup$ as you suggested, I tried to remove a variable at a time in the logistic regression. About half of the variables when they are removed (one at a time), the residual deviance shoots up to more than 4000. Does this mean that these removed variables are important? $\endgroup$ – lll Sep 18 '18 at 23:10
0
$\begingroup$

You likely have too many predictors for your model. A rule of thumb is that you should have 10 of each outcome (1/0, Y/N) for each predictor in your model. So for example if you have 100 observations and for 30 of your outcomes y=1, then you should only include 3 predictors in your model. Any more and you run the risk what you are seeing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.