Test GLM model using null and model deviances

I've built a glm model in R and have tested it using a testing and training group so am confident it works well. The results from R are:

Coefficients:
Estimate Std. Error  t value Pr(>|t|)
(Intercept)               -2.781e+00  1.677e-02 -165.789  < 2e-16 ***
Coeff_A                    1.663e-05  5.438e-06    3.059  0.00222 **
log(Coeff_B)               8.925e-01  1.023e-02   87.245  < 2e-16 ***
log(Coeff_C)              -3.978e-01  7.695e-03  -51.689  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for quasibinomial family taken to be 0.9995149)

Null deviance: 256600  on 671266  degrees of freedom
Residual deviance: 237230  on 671263  degrees of freedom
AIC: NA

All the p values for the coefficients are small as expected.

Looking at this question (Interpreting Residual and Null Deviance in GLM R), I should be able to calculate if the null hypothesis holds using the following equation:

p-value = 1 - pchisq(deviance, degrees of freedom)

Sticking this in gives:

1 - pchisq(256600, 671266)
[1] 1

So am I correct in thinking the null hypothesis cannot be rejected here, even though the p values for all coefficients are so small or have I misinterpreted how to calculate this?

1 Answer

There is a misunderstanding here. The difference between the null deviance and the model's deviance is distributed as a chi-squared with degrees of freedom equal to the null df minus the model's df. For your model, that would be:

1-pchisq(256600 - 237230, df=(671266 - 671263))
# [1] 0

By default, pchisq() gives the proportion of the distribution to the left of the value. To get the proportion more extreme than your difference, you can specify lower.tail = FALSE or subtract the result from $1$ (as you and I have done).

• What hypothesis exactly are you testing with the statement 1-pchisq(256600 - 237230, df=(671266 - 671263))? – jII Apr 12 '15 at 17:30
• @jesterII, you are checking if the deviance changed more than might be expected by chance. Ie, you are testing if the model as a whole is better than the null model. It is analogous to the global F test in a linear model. – gung - Reinstate Monica Apr 12 '15 at 17:50
• The null hypothesis is 'the model as a whole is better than the null model', and you have rejected the null hypothesis, which means the model is poor? – jII Apr 12 '15 at 17:55
• @jesterII, no the null hypothesis is: 'the model as a whole is no better than the null model'. Since this has been rejected, we conclude that the data are not consistent with the null model. NB, this does not necessarily mean that our model is 'good' or 'correct'. – gung - Reinstate Monica Apr 12 '15 at 17:57