I am doing a multiple regression analysis and my focus is finding the best set of independent variables for prediction. I am starting to know my dataset and the behavior of each variable. I am doing a residuals analysis (with a great help of R) and my question is about the meaning of the residual plot for one of these variables.

Studentized residual plot

Can I say that these variable residuals are a Null Plot kind with some outliers? Moreover, what does it help for my goal of finding good variables?

enter image description here

  • 1
    $\begingroup$ Using decimal numbers to plot the residuals creates an unreadable and possibly biased clutter throughout key parts of this plot. It would help if you would recreate it using small, uniformly sized point symbols instead. $\endgroup$ – whuber Sep 22 '14 at 21:02
  • 1
    $\begingroup$ As noted in several answers below, your model may be mis-specified. Can you say more about what your data are? What do they represent & where do they come from? What are your goals for the model? $\endgroup$ – gung - Reinstate Monica Sep 22 '14 at 21:14
  • $\begingroup$ @whuber, I added a cleaner plot. $\endgroup$ – cctruc Sep 22 '14 at 21:53
  • 3
    $\begingroup$ I wondered if something like that might be the case. So your response variable is a count & you are using standard OLS (linear) regression. You need to use a Poisson (etc) GLiM instead. $\endgroup$ – gung - Reinstate Monica Sep 22 '14 at 21:56
  • 3
    $\begingroup$ FWIW, the standard variance-stabilizing transformation for count data is the square root. Old-timers like me used to do that in the days before all this GLIM stuff was available. Maybe not as good as Poisson regression, but possibly better than no transformation. From the new plot (and the explanation of the standardized residuals), I can easily see two more diagonal rows of points, I bet you have quite a few 1s and 2s in the data also. These diagonals appear because $e = y - \hat y$, so for fixed $y$, we have a line with intercept $y$ and slope $-1$. $\endgroup$ – Russ Lenth Sep 23 '14 at 0:30

Maybe -- but it does also have some characteristics of the horn-shaped plot you get when a transformation might help. Are these ordinary residuals, or some kind of standardized ones?

The reason I ask is that it's not unusual to see a downward-sloping edge in a residuals-vs-predicted plot; it happens when there is a frequently-attained lower bound (e.g., zero) on the $y$ values. However, if that is the case, that lower edge should have a slope of $-1$ and the slope in the plot is more like $-0.1$. But if the residuals are standardized, that'd explain it.

You can use Tukey's nonadditivity test to see if a transformation might help. The technique is as follows:

  1. Obtain the predicted values, $\hat y_i$
  2. Compute the variable $N$ with values $N_i = \hat y_i^2$
  3. Fit the same model with $N$ as an additional predictor
  4. If the $t$ statistic for $N$ is significant (this is the test of Tukey's one d.f. for nonadditivity), it suggests that a transformation of the response might help. As a rough estimate, use $y$ raised to the $1-\hat\beta_N$ power, or $\log y$ if this is nearly zero.

Note: This is only for diagnostic purposes. Don't include $N$ in your final model, or in any steps along the way! Another note: A similar idea is the Atkinson score test, where you use $N_i = \hat y_i\log\hat y_i$

An additional suggestion is to plot residuals against everything you can think of (time order, predictors in the model, predictors not in the model) to see if there is any kind of apparent pattern in those.

And one more comment: Sometimes, a bad residual plot is good news! A really poor-fitting model often has a nice residual plot but doesn't predict the response worth a darn. When the residual plot starts looking bad, it can mean that you've explained enough of the variations in the response that you can now see the more minor defects in the model.

| cite | improve this answer | |
  • 2
    $\begingroup$ +1, this is a really great answer. Here is where I think we see the wisdom that comes from years of experience & expertise. $\endgroup$ – gung - Reinstate Monica Sep 22 '14 at 21:10
  • 1
    $\begingroup$ Yes, the residuals are standardized (studentized). Sorry for not having commented that.. There are many 0 values for y as well! I didn't know about the Tukey's nonadditivity. I will try! Thank you! $\endgroup$ – cctruc Sep 22 '14 at 22:00

A residual plot (meaning, residuals on one axis, conventionally the $y$ axis) and predicted or fitted on the other, conventionally the $x$ axis) is a kind of overall health check on a regression-type model.

Unless this plot is mislabelled it doesn't refer to a particular "independent variable", nor can it help much in determining which independent variables to use in a model. I am with those who think that "independent variable" is an outdated and unsatisfactory term; unfortunately we can't all agree on which term is best, but "predictor" or "covariate" suits many people better.

The main message from this plot that I pick up is to note that residuals all lie above a line with negative slope. The key question to answer is whether the response is positive by definition, or at least non-negative. If so, negative predicted values don't make much sense, and a logarithmic transformation or using a logarithmic link seems strongly indicated. I would want a more symmetric and more nearly patternless scatter to declare the regression satisfactory.

On that and on other grounds I tend to disagree with @user1669710. Experienced users of statistics are typically reluctant to declare data points as outliers without very strong reasons, particularly whenever heteroscedasticity is a more plausible explanation. In particular, the appearance of outliers is often illusive and is not maintained when using an appropriate non-linear scale.

If this is a plot from a standard regression, the mean residual will be zero, period, and is not, and cannot be, diagnostic.

EDIT: As the response variable has now been explained as non-negative, a Poisson regression is strongly indicated. Note that (raw) residuals are now expected to be heteroscedastic, but standardization can help with that.

| cite | improve this answer | |
  • $\begingroup$ Thank you for the answer! How can I have a more symmetric and more nearly patternless scatter? $\endgroup$ – cctruc Sep 22 '14 at 22:04
  • $\begingroup$ As @gung suggests, you should think of a more appropriate model. Poisson regression sounds like the next thing to try. $\endgroup$ – Nick Cox Sep 22 '14 at 22:45

Residual plots are used to identify heteroscedasticity, non-linear dependence and outliers. If the data is heteroscedastic you will see a lot of variation along the horizontal axis in the vertical spread of the errors. In your case, the difference in the vertical spread is looking more like outliers than heteroscedasticity, however, I am a bit unsure about the 100-150 range, which looks like the spread is slightly different. Non-linear dependence typically has more structured plots than a "blob" like yours. In your case, looks like the regression you are using is an "ok" fit to the data. I'd also look at the residual mean, which if not zero, indicates an inferior fit of the model to the data.

| cite | improve this answer | |
  • 1
    $\begingroup$ Did you notice that the response variable appears to have a floor (probably 0) under it? That leads to obvious heteroscedasticity and bias in the residuals at low predicted values. A different regression model altogether is called for here. $\endgroup$ – whuber Sep 22 '14 at 21:00
  • $\begingroup$ @whuber, I think the residual mean is guaranteed to be zero as long as the intercept is in the model. $\endgroup$ – Russ Lenth Sep 22 '14 at 21:03
  • $\begingroup$ @rvl When a constant term is included in least squares regression, that is correct: but it still does not preclude the presence of a local bias in the residuals, which is quite apparent here where negative values are predicted. I think this would become more evident with a better plot of the results; this one does not show what's really going on in any detail. $\endgroup$ – whuber Sep 22 '14 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.