3
$\begingroup$

I'm having some trouble interpretting whether the model is a good fit. Below is an extract from some output from R.

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 3946.2  on 2999  degrees of freedom
Residual deviance: 3924.3  on 2991  degrees of freedom
AIC: 3942.3

Number of Fisher Scoring iterations: 4

> 1 - pchisq(mylogit$null.deviance - mylogit$deviance, mylogit$df.null - mylogit$df.residual)
[1] 0.004985122

So, as the p-value is low, this would indicate that model is a good fit ie. it is significant right ?

Many thanks

$\endgroup$
1
  • 3
    $\begingroup$ "Good fit" is not necessarily synonymous with "significant", -- perhaps oddly, this can be the case even when your significance test is called a goodness of fit. $\endgroup$ – Glen_b Jan 12 '14 at 21:02
4
$\begingroup$

The test is telling you roughly that your model is 'better than nothing'. That is: it is a significantly better fit to the data than a model with no covariates in it at all. It might also be a 'good fit' in other terms, but you'd have to say what 'good' means for you. But whatever it means, it probably isn't usefully operationalised as being better than an empty model, although you might expect it to at least be that.

In passing, note that R's anova function will do some more detailed deviance testing for you that you might find useful (with in my experience, rather better numerical behaviour). For example,

anova(mylogit, test='Chisq')

will tell you about the consequences of adding higher level terms. Moreover, you can compare any number of models this way, not just your final with the null model. See ?anova.glm for details.

One possibly more useful way to see how good the fit it is to compare the model to the data rather than another model. If your dependent variable is binary then you can pick a threshold for fitted probabilities and cross-tabulate them with it. Something like

thresh <- 0.5 ## if 0->1 and 1->0 errors are equally costly
fitted.probs <-  predict(mylogit, type='response')    
preds <- ifelse(fitted.probs > thresh, 1, 0)
table(preds, Y) ## assuming Y is the dependent variable

A more thorough approach would be to plot the ROC curve (essentially this procedure but for all possible thresholds), for which there are R packages available.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.