26
$\begingroup$

Can anyone tell me how to interpret the 'residuals vs fitted', 'normal q-q', 'scale-location', and 'residuals vs leverage' plots? I am fitting a binomial GLM, saving it and then plotting it.

$\endgroup$
  • 3
    $\begingroup$ Do you know how to interpret those plots when dealing with a regular linear regression? Because that should be your starting point. $\endgroup$ – Steve S Oct 26 '14 at 17:56
47
$\begingroup$

R does not have a distinct plot.glm() method. When you fit a model with glm() and run plot(), it calls ?plot.lm, which is appropriate for linear models (i.e., with a normally distributed error term).

In general, the meaning of these plots (at least for linear models) can be learned in various existing threads on CV (e.g.: Residuals vs. Fitted; qq-plots in several places: 1, 2, 3; Scale-Location; Residuals vs Leverage). However, those interpretations are not generally valid when the model in question is a logistic regression.

More specifically, the plots will often 'look funny' and lead people to believe that there is something wrong with the model when it is perfectly fine. We can see this by looking at those plots with a couple of simple simulations where we know the model is correct:

  # we'll need this function to generate the Y data:
lo2p = function(lo){ exp(lo)/(1+exp(lo)) }

set.seed(10)                    # this makes the simulation exactly reproducible
x  = runif(20, min=0, max=10)   # the X data are uniformly distributed from 0 to 10
lo = -3 + .7*x                  # this is the true data generating process
p  = lo2p(lo)                   # here I convert the log odds to probabilities
y  = rbinom(20, size=1, prob=p) # this generates the Y data

mod = glm(y~x, family=binomial) # here I fit the model
summary(mod)                    # the model captures the DGP very well & has no
# ...                           #  obvious problems:
# Deviance Residuals: 
#      Min        1Q    Median        3Q       Max  
# -1.76225  -0.85236  -0.05011   0.83786   1.59393  
# 
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)  
# (Intercept)  -2.7370     1.4062  -1.946   0.0516 .
# x             0.6799     0.3261   2.085   0.0371 *
# ...
# 
# Null deviance: 27.726  on 19  degrees of freedom
# Residual deviance: 21.236  on 18  degrees of freedom
# AIC: 25.236
# 
# Number of Fisher Scoring iterations: 4

Now lets look at the plots we get from plot.lm():

enter image description here

Both the Residuals vs Fitted and the Scale-Location plots look like there are problems with the model, but we know there aren't any. These plots, intended for linear models, are simply often misleading when used with a logistic regression model.

Let's look at another example:

set.seed(10)
x2 = rep(c(1:4), each=40)                    # X is a factor with 4 levels
lo = -3 + .7*x2
p  = lo2p(lo)
y  = rbinom(160, size=1, prob=p)

mod = glm(y~as.factor(x2), family=binomial)
summary(mod)                                 # again, everything looks good:
# ...
# Deviance Residuals: 
#   Min       1Q   Median       3Q      Max  
# -1.0108  -0.8446  -0.3949  -0.2250   2.7162  
# 
# Coefficients:
#                Estimate Std. Error z value Pr(>|z|)    
# (Intercept)      -3.664      1.013  -3.618 0.000297 ***
# as.factor(x2)2    1.151      1.177   0.978 0.328125    
# as.factor(x2)3    2.816      1.070   2.632 0.008481 ** 
# as.factor(x2)4    3.258      1.063   3.065 0.002175 ** 
# ... 
# 
# Null deviance: 160.13  on 159  degrees of freedom
# Residual deviance: 133.37  on 156  degrees of freedom
# AIC: 141.37
# 
# Number of Fisher Scoring iterations: 6

enter image description here

Now all the plots look strange.

So what do these plots show you?

  • The Residuals vs Fitted plot can help you see, for example, if there are curvilinear trends that you missed. But the fit of a logistic regression is curvilinear by nature, so you can have odd looking trends in the residuals with nothing amiss.
  • The Normal Q-Q plot helps you detect if your residuals are normally distributed. But the deviance residuals don't have to be normally distributed for the model to be valid, so the normality / non-normality of the residuals doesn't necessarily tell you anything.
  • The Scale-Location plot can help you identify heteroscedasticity. But logistic regression models are pretty much heteroscedastic by nature.
  • The Residuals vs Leverage can help you identify possible outliers. But outliers in logistic regression don't necessarily manifest in the same way as in linear regression, so this plot may or may not be helpful in identifying them.

The simple take home lesson here is that these plots can be very hard to use to help you understand what is going on with your logistic regression model. It is probably best for people not to look at these plots at all when running logistic regression, unless they have considerable expertise.

$\endgroup$
  • 4
    $\begingroup$ Good grief this is an amazing response. $\endgroup$ – d8aninja Nov 10 '15 at 2:11
  • $\begingroup$ @gung Could you please say what can we do if it's a glm? Is there an alternative to Q-Q, Residual vs Fitted? For example I plotted a poisson link GLM, but I don't know how to analyze if it's good fit $\endgroup$ – GRS Nov 9 '16 at 18:53
  • 1
    $\begingroup$ @GRS, try reading this. $\endgroup$ – gung Nov 9 '16 at 19:11
0
$\begingroup$
  1. Residuals vs fitted - there should be no strong patterns (mild patterns are not a problem, see @gung's answer) and no outliers, residuals should be randomly distributed around zero.
  2. Normal Q-Q - residuals should go around the diagonal line, i.e. should be normally distributed (see wiki for Q-Q plot). This plot helps checking if they are approximately normal.
  3. Scale-location - as you can see, on Y axis there are also residuals (like in Residuals vs fitted plot), but they are scaled, so it's similar to (1), but in some cases it works better.
  4. Residuals vs Leverage -- it helps to diagnose outlying cases. As in previous plots, outlying cases are numbered, but on this plot if there are any cases that are very different from the rest of the data they are plotted below thin red lines (check wiki on Cook's distance).

Read more on assumptions of regression as in many aspects there are similar (e.g. here, or tutorial on regression in R here).

$\endgroup$
  • 2
    $\begingroup$ This response is incorrect in the context of glm, as well laid out by @gung , in the response above. If you are looking for an answer to this question, IGNORE THIS RESPONSE. If you are the author of this response, check out the answer above by gung. If you agree with, you should consider deleting this response, as it is misleading. $\endgroup$ – colin Mar 17 '16 at 18:07
  • $\begingroup$ @colin would you like to comment what exactly is wrong in your opinion with this answer? The only difference between mine and the second answer is that gung goes into more details... $\endgroup$ – Tim Mar 17 '16 at 19:36
  • 3
    $\begingroup$ you describe how these plots should be used in the context of linear regression. gung describes why these interpretations fail in this case, because they are being applied to a binomial glm model. So, if a user interpreted these diagnostic plots as you suggest (and your suggestions would be helpful in a case of lm), they will erroneously conclude that their model violates the assumptions of glm, when in reality it does not. $\endgroup$ – colin Mar 17 '16 at 19:47
  • $\begingroup$ you write, "Residuals vs fitted - there should be no patterns and no outliers, residuals should be randomly distributed around 0." gung writes, "The Residuals vs Fitted plot can help you see, for example, if there are curvilinear trends that you missed. But the fit of a logistic regression is curvilinear by nature, so you can have odd looking trends in the residuals with nothing amiss." ...one of you must be wrong. $\endgroup$ – colin Mar 17 '16 at 20:23
  • 2
    $\begingroup$ I agree, this is not black and white, but there are many instances where there ARE strong trends, but the model specified is totally appropriate under the assumptins of the specific glm. Therefore, asserting, "there should be no patterns and no outliers, residuals should be randomly distributed around 0." is easily interpreted that if there is a pattern, you have violated model assumptions. This is not the case. $\endgroup$ – colin Mar 17 '16 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.