81
$\begingroup$

For linear regression, we can check the diagnostic plots (residuals plots, Normal QQ plots, etc) to check if the assumptions of linear regression are violated.

For logistic regression, I am having trouble finding resources that explain how to diagnose the logistic regression model fit. Digging up some course notes for GLM, it simply states that checking the residuals is not helpful for performing diagnosis for a logistic regression fit.

Looking around the internet, there also seems to be various "diagnosis" procedures, such as checking the model deviance and performing chi-squared tests, but other sources state that this is inappropriate, and that you should perform a Hosmer-Lemeshow goodness of fit test. Then I find other sources that state that this test may be highly dependent on the actual groupings and cut-off values (may not be reliable).

So how should one diagnose the logistic regression fit?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Possible duplicate (or special case) of stats.stackexchange.com/questions/29271/… or stats.stackexchange.com/questions/44643/…, although neither of them have answers that will really solve it for you. $\endgroup$
    – Peter Ellis
    Commented Dec 4, 2012 at 0:52
  • 1
    $\begingroup$ I recommend you read Scott Menard's monograph, which not too long ago was available in its entirety for free on the web. $\endgroup$
    – rolando2
    Commented Dec 4, 2012 at 1:27
  • 2
    $\begingroup$ This question on goodness of fit measures for logistic regression may be helpful (although goodness of fit is of course only a small part of model diagnostics): stats.stackexchange.com/questions/3559/logistic-regression-which-pseudo-r-squared-measure-is-the-one-to-report-cox/3570 $\endgroup$
    – Stephan Kolassa
    Commented Dec 4, 2012 at 11:22
  • $\begingroup$ Hosmer-Lemeshow is considered obsolete: stats.stackexchange.com/questions/273966/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented May 14, 2020 at 12:15

3 Answers 3

Reset to default
41
$\begingroup$

A few newer techniques I have come across for assessing the fit of logistic regression models come from political science journals:

  • Greenhill, Brian, Michael D. Ward & Audrey Sacks. 2011. The separation plot: A new visual method for evaluating the fit of binary models. American Journal of Political Science 55(4):991-1002.
  • Esarey, Justin & Andrew Pierce. 2012. Assessing fit quality and testing for misspecification in binary-dependent variable models. Political Analysis 20(4): 480-500. Preprint PDF Here

Both of these techniques purport to replace Goodness-of-Fit tests (like Hosmer & Lemeshow) and identify potential mis-specification (in particular non-linearity in included variables in the equation). These are particularly useful as typical R-square measures of fit are frequently criticized.

Both of the above papers above utilize predicted probabilities vs. observed outcomes in plots - somewhat avoiding the unclear issue of what is a residual in such models. Examples of residuals could be contribution to the log-likelihood or Pearson residuals (I believe there are many more though). Another measure that is often of interest (although not a residual) are DFBeta's (the amount a coefficient estimate changes when an observation is excluded from the model). See examples in Stata for this UCLA page on Logistic Regression Diagnostics along with other potential diagnostic procedures.

I don't have it handy, but I believe J. Scott Long's Regression Models for Categorical and Limited Dependent Variables goes in to sufficient detail on all of these different diagnostic measures in a simple manner.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ There are bucketloads of other books on (at least in parts if not the entirety) for logistic regression. Agresti's various categorical data analysis text books, Scott Menard, Hosmer and Lemeshow, and Frank Harrell's RMS book are all ones I have seen recommended on this forum by various contributors. $\endgroup$
    – Andy W
    Commented Dec 4, 2012 at 15:19
  • $\begingroup$ Thank you for your answer. I guess there is no simple answer to my question. I will take a look at your recommendations. Cheers. $\endgroup$
    – ialm
    Commented Dec 5, 2012 at 22:56
25
$\begingroup$

The question was not well enough motivated. There has to be a reason to run model diagnostics, such as

  • Potential to change the model to make it better
  • Not knowing which directed tests to use (i.e., tests of non-linearity or interaction)
  • Failing to grasp that changing the model can easily distort statistical inference (standard errors, confidence intervals, $P$-values)

Except for checking things that are orthogonal to the algebraic regression specification (e.g., examining the distribution of residuals in ordinary linear models), model diagnostics can create as many problems as they solve in my opinion. This is especially true of the binary logistic model since it has no distributional assumption.

So it is usually better to spend time specifying the model, especially to not assume linearity for variables thought to be strong for which no prior evidence suggests linearity. In some occasions you can pre-specify a model that must fit, e.g., if the number of predictors is small or you allow all predictors to be nonlinear and (correctly) assume no interactions.

Anyone feeling that model diagnostics can be used to change the model should run that process within a bootstrap loop to correctly estimate the induced model uncertainties.

Improve this answer
$\endgroup$
7
  • 6
    $\begingroup$ I agree that model diagnostics should stem from the purpose of the modelling. However, I got the impression from your opening paragraph that you think we shouldn't be checking the models that we fit to data. I am pretty sure this is not what you had in mind. Additionally, the binary logistic model most certainly does have distributional assumptions! (most obvious being that only two values exist for the response) $\endgroup$
    – probabilityislogic
    Commented Aug 19, 2016 at 13:02
  • 4
    $\begingroup$ Other than assuming there are only 2 specific possible values for Y, the binary logistic model has no dist. assumptions. I personally don't use diagnostic plots with logistic regression very often, opting instead to specify models that are flexible enough to fit the data in any way the sample size gives us the luxury to examine. In OLS the main diagnostic plot I use is the qq plot for normality of residuals. $\endgroup$
    – Frank Harrell
    Commented Aug 19, 2016 at 20:17
  • $\begingroup$ From a generalized linear model perspective, the logistic model arise from the binomial distribution (Bernoulli distribution). But even then, it is har to interpret residuals. $\endgroup$
    – New_to_this
    Commented Nov 1, 2016 at 7:42
  • $\begingroup$ Talking about distributions when the random variable can take on only two values (i.e., Bernoulli distribution) is not helpful because there is no way that the distributional assumption can go wrong unless the observations are not independent. $\endgroup$
    – Frank Harrell
    Commented Nov 1, 2016 at 12:39
  • 5
    $\begingroup$ @FrankHarrell I realize that you know what you're talking about here, but I don't think it will be clear to the entire community from your post/comments that mis-specification of the linear predictor (or even the additive predictor in a GAM framework) can cause problems for logistic regression. Many might not appreciate that this creates non-independence in the data, as you say. When students first encounter linear regression, they learn to inspect the residuals without distinguishing between misspecification of the linear predictor and misspecification of the error distribution. $\endgroup$
    – Jacob Socolar
    Commented Jan 26, 2017 at 3:54
11
$\begingroup$

This thread is quite old, but I thought it would be useful to add that, since recently, you can use the DHARMa R package to transform the residuals of any GL(M)M into a standardized space. Once this is done, you can visually assess / test residual problems such as deviations from the distribution, residual dependency on a predictor, heteroskedasticity or autocorrelation in the normal way. See the package vignette for worked-through examples, also other questions on CV here and here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.