I am carrying out a logistic regression with $24$ independent variables and $123,996$ observations. I am evaluating the model fit in order to determine if the data meet the model assumptions and have produced the following binned residual plot using the arm R package:

enter image description here

Obviously there are some bad signs in this plot: many points fall outside the confidence bands and there is a distinctive pattern to the residuals. My question is - can I attach these issues to specific assumptions of the logistic regression model? For instance, can I say that there is evidence of non-linearity in the independent variables or of heteroscedasticity? If not, are there other diagnostics I can produce to help identify where the problem lies?


Based on Daniel's answer, it appears that the main issue is I was using residuals on the logit scale but expected values on the response scale. If I reproduce the plot with the residuals also on the response scale it looks like this:

enter image description here

which is much more believable.

  • Please describe the statistical theory that implies that such a residual plot is useful. – Frank Harrell May 21 '14 at 19:16
  • @FrankHarrell See discussion of Figure 17 in Gelman et al (2000) "Diagnostic checks for discrete data regression models using posterior predictive simulations" - available here: stat.columbia.edu/~gelman/research/published/dogs.pdf. Also page 97 of Andrew Gelman and Jennifer Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models, Cambridge University Press – M. Berk May 21 '14 at 19:39
  • Can you summarize what exactly you are attempted to do with such plots? For binary logistic regression there is no distributional assumption, and for regression assumptions it's best to just fit the model flexibly (regression splines, etc.) or to use traditional partial residual plots. – Frank Harrell May 21 '14 at 21:57
  • @FrankHarrell I've edited the question to clarify that I'm trying to assess whether the data meet the model assumptions. Thanks for the introduction to partial residual plots, I think these are exactly what I'm looking for. – M. Berk May 22 '14 at 8:25
up vote 4 down vote accepted
+50

Either I am misinterpreting your plot or there is some problem. The fact that you have negative residuals for near 0 expected values implies that your model is predicting negative value. This should not be possible for logistic regression models which only predict in the (0, 1) interval, unless you are using the log-odds output of the model in which case residual error should be undefined. Since logistic regression is a classification method, it is more useful to look at the confusion matrix first. You should also specify if the graph is based on the train data or a separate test set.

  • I believe you have correctly identified the issue. I've got the residuals on the logit scale and the fitted values on the response scale (i.e. between 0 and 1). I've reproduced the plot with the residuals on the response scale and it looks much more believable. – M. Berk May 22 '14 at 8:30

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.