9
$\begingroup$

I am carrying out a logistic regression with $24$ independent variables and $123,996$ observations. I am evaluating the model fit in order to determine if the data meet the model assumptions and have produced the following binned residual plot using the arm R package:

enter image description here

Obviously there are some bad signs in this plot: many points fall outside the confidence bands and there is a distinctive pattern to the residuals. My question is - can I attach these issues to specific assumptions of the logistic regression model? For instance, can I say that there is evidence of non-linearity in the independent variables or of heteroscedasticity? If not, are there other diagnostics I can produce to help identify where the problem lies?

Based on Daniel's answer, it appears that the main issue is I was using residuals on the logit scale but expected values on the response scale. If I reproduce the plot with the residuals also on the response scale it looks like this:

enter image description here

which is much more believable.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Please describe the statistical theory that implies that such a residual plot is useful. $\endgroup$
    – Frank Harrell
    May 21, 2014 at 19:16
  • 1
    $\begingroup$ @FrankHarrell See discussion of Figure 17 in Gelman et al (2000) "Diagnostic checks for discrete data regression models using posterior predictive simulations" - available here: stat.columbia.edu/~gelman/research/published/dogs.pdf. Also page 97 of Andrew Gelman and Jennifer Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models, Cambridge University Press $\endgroup$
    – M. Berk
    May 21, 2014 at 19:39
  • $\begingroup$ Can you summarize what exactly you are attempted to do with such plots? For binary logistic regression there is no distributional assumption, and for regression assumptions it's best to just fit the model flexibly (regression splines, etc.) or to use traditional partial residual plots. $\endgroup$
    – Frank Harrell
    May 21, 2014 at 21:57
  • $\begingroup$ @FrankHarrell I've edited the question to clarify that I'm trying to assess whether the data meet the model assumptions. Thanks for the introduction to partial residual plots, I think these are exactly what I'm looking for. $\endgroup$
    – M. Berk
    May 22, 2014 at 8:25
  • $\begingroup$ Could you show the code used to produce each plot so that we can see what you changed between the two plots? $\endgroup$
    – canderson156
    Jul 16, 2020 at 14:42

2 Answers 2

Reset to default
4
+50
$\begingroup$

Either I am misinterpreting your plot or there is some problem. The fact that you have negative residuals for near 0 expected values implies that your model is predicting negative value. This should not be possible for logistic regression models which only predict in the (0, 1) interval, unless you are using the log-odds output of the model in which case residual error should be undefined. Since logistic regression is a classification method, it is more useful to look at the confusion matrix first. You should also specify if the graph is based on the train data or a separate test set.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I believe you have correctly identified the issue. I've got the residuals on the logit scale and the fitted values on the response scale (i.e. between 0 and 1). I've reproduced the plot with the residuals on the response scale and it looks much more believable. $\endgroup$
    – M. Berk
    May 22, 2014 at 8:30
0
$\begingroup$

For others in similar situations, I might suggest looking at the simulation-based residuals for GLMs (and GAMs, and GLMMs) available in the DHARMa package for R. These to me look more theoretically justified than the binned residual plots, and come with easily implemented functions. More here: https://cran.r-project.org/web/packages/DHARMa/vignettes/DHARMa.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.