I used a least squares method on my data set, using the lsqr Matlab function. I know that the norm of residuals is a measure of the goodness of fit, but how can I assess whether the value of the norm of residuals is "good"? Which is the range of this measure?

  • $\begingroup$ You need to read a regression book or take a regression class at your local university. A page of discussion on CV simply won't help you. $\endgroup$ – StasK Sep 16 '11 at 14:14
  • 6
    $\begingroup$ @Stas That advice seems unnecessarily pessimistic. $\endgroup$ – whuber Sep 16 '11 at 14:36
  • $\begingroup$ @whuber, a person who would have taken a regression class would not have asked this question, and would refer to this quantity as sum of squared residuals rather than the norm (which is a very reasonable term in the geometry of least squares, of course). What Charlie does below is exactly what I was afraid would happen: introducing $R^2$ as the "standardized" norm, with all the discussion around it (first we teach how to use it to select the best model, and then we teach why it should not be used in this capacity). $\endgroup$ – StasK Sep 16 '11 at 15:16
  • 5
    $\begingroup$ @Stas It's not a good idea to make inferences about the person based on the question they ask. Sure, how a question is posed can indicate whether a technical or non-technical answer would be appropriate. But it shouldn't ever lead us to believe that the proposer is incapable of learning from an answer or not using it effectively. In this case, on the contrary, "norm" suggests a relatively strong mathematical background and the fact that the proposer is concerned about goodness of fit suggests an awareness of important and subtle issues. $\endgroup$ – whuber Sep 16 '11 at 15:56
  • $\begingroup$ You can find a nice intuitive explanation about residual norms in the link below shodor.org/interactivate/discussions/UsingResiduals $\endgroup$ – user76461 May 6 '15 at 21:33

Residuals are "acceptable" when they have, at least approximately, the following characteristics:

  • They are not associated with the fitted values (there's no evident trend or relationship between them).

  • They are centered around zero.

  • Their distribution is symmetric.

  • They contain no, or extremely few, unusually large or small values ("outliers").

  • They are not correlated with other variables you have in the data set.

These are not criteria; they're guidelines. For instance, correlation with other variables is sometimes ok. The correlation merely suggests that the residuals could be further improved by including those variables in the model. But the first three points are as close to criteria as we can get in general, in the sense that a strong violation of any of the them is a clear indication the model is wrong.

There are plenty of examples of evaluating residuals and goodness of fit in postings on this site; one recent one where the residuals look acceptable appears at Looking for estimates for my data using cumulative beta distribution. An example of clearly unacceptable residuals appears (inter alia) in the question at Testing homoscedasticity with Breusch-Pagan test. There, the distribution of residuals is asymmetric (there is a long tail in the negative range), the residuals vary in important ways with another variable (the "index"), and they exhibit a v-shaped association with the fitted values.

A set of acceptable residuals is "good" when their typical size is small enough to alleviate any worries that your conclusions might be incorrect. "Small enough" depends on how the model will be used, but the main point is that you want to pay attention to how large the residuals can be, because that measures the typical deviation between the dependent variable and the fit. When your data are representative of a process or population, that typical deviation estimates how closely the model will predict the unsampled members of the population.

For example, a model of survivorship from a medical procedure might express its residuals as percentage of survival time. If a typical size is 100%, the model may be almost worthless ("you might die any time between tomorrow and 20 years from now) but if it's 10%, the model is probably excellent for anybody ("people with your condition usually live between 9 and 11 years"). A 1 km residual in a spatial location model would be great for sending a satellite to Mars but could shipwreck boats in a harbor. Context matters when evaluating the goodness of fit.

Several measures of residual size are in use, again depending on the purpose of the analysis. The commonest is an (adjusted) root mean square, which is almost always reported by least squares software. It would be naive and foolhardy to rely on this number without checking the guidelines for acceptability. Once you have confirmed the residuals are acceptable, though, this number is an effective way to evaluate and report the goodness of fit.

Among the alternative measures of residual size, an excellent one is the "H-spread" of the residuals. (Split the set of residuals into an upper half and lower half. The H-spread is the difference between the median of the upper half and the median of the lower half. It is practically the same as the interquartile range of the residuals). This measure is not as sensitive to the most extreme residuals as the root mean square. Nevertheless, as indicated in the guideline about outliers, it's also a good idea to look at the sizes of the most positive and most negative residuals.

Because it is mentioned elsewhere in this thread, let's look at $R^2$. This figure is related to the size of residuals, but the relation is indirect. As you can see from the formula, it depends on the total variation in the dependent variable, which in turn depends on how much the independent variables vary in the dataset. This makes it much less useful than directly examining the root mean square residual. $R^2$ values can also be erroneous: they can become extremely high due to the presence of even a single "high leverage" value in the data, giving a false impression of a good fit. Any decent measure of the typical residual size will not have this problem.

  • 2
    $\begingroup$ (+1) The paragraph on "acceptable residuals" is very well-put. $\endgroup$ – chl Sep 17 '11 at 9:25

Imagine that we didn't model anything and we didn't have any information on $x$ to use in our predictions. What would our guess of $y$ be?

If we performed the following regression: $$ \begin{align*} y_i = \beta_0 + \epsilon_i \end{align*}$$ then $\hat{\beta}_0 = \bar{y}$.

We can write $$ \begin{align*} y_i = \hat{y}_i + e_i, \end{align*}$$ where $e$ is the residual. Now, subtract $\bar{y}$ from both sides: $$ \begin{align*} y_i - \bar{y} = \hat{y}_i - \bar{y} + e_i. \end{align*}$$ Let's square both sides and sum over all $i$: $$ \begin{align*} \sum{\left(y_i - \bar{y}\right)^2} = \sum{\left(\hat{y}_i - \bar{y}\right)^2} + \sum{e_i^2} \end{align*}$$ (we used the fact that $\sum{\left(\hat{y}_i - \bar{y}\right)e_i} = 0$). }

We create a statistic called $R^2$: $$ \begin{align*} R^2 = \frac{\sum{\left(\hat{y}_i - \bar{y}\right)^2}}{\sum{\left(y_i - \bar{y}\right)^2}} = 1 - \frac{\sum{e_i^2}}{\sum{\left(y_i - \bar{y}\right)^2}}. \end{align*}$$ This is the ratio of the variation predicted by our model to the variation not predicted by the mean alone, the simplest possible model of our outcome. Of the variation to be predicted, what fraction is predicted by your model? Alternatively, it is 1 minus the variation that remains relative to the total variation.

Since OLS minimizes the sum of squared residuals, for any given data $y$, you can view OLS as choosing coefficients to maximize the $R^2$.

Because our models include an intercept, the regression line goes through the point of means ($\bar{x}, \bar{y}$) and thus they predict no less than the mean alone.

Using the derivation above and the fact that our regression includes an intercept term, $R^2$ is bounded between 0 and 1.

Additionally, $R^2$ is equal to the sample correlation squared $\left(r_{x,y} \text{ for univariate regression and }r_{\hat{y},y} \text{ generally}\right)$.

$R^2$ is the most common goodness-of-fit measure for regression, but it needs to come with a warning siren.

All that $R^2$ tells you is the proportion of the overall variation that is predicted by your model. Note that I avoid the typical "is explained by your model" phrasing that people use. This implies causality and $R^2$ does not measure that.

You can have a perfect model: $y$ is just its mean plus some error---that's the definition of $y$---but, the $R^2$ is 0. Your model is exactly right, but it has a small $R^2$ simply because you can't predict the variation of $y$ no matter what information (variables) you use.

On the other hand, you can have a really high $R^2$, but have a completely wrong model when two things are correlated for some other reason. A time series example is if I regress my weight from ages 0 to 27 on the weight of someone else from ages 0 to 27. We'll get a high $R^2$ simply because our weights were trending up for a while as we grew and relatively plateaued later on; my weighted was not caused by or related to in any way the weight of the other person (see this question on spurious causation).

  • $\begingroup$ (-1) It is strange that your answer proposes $R^2$, but fails to relate it to the residuals and then devotes itself to explaining why it shouldn't be used! (This ground has been covered in the thread at stats.stackexchange.com/questions/13314/…) $\endgroup$ – whuber Sep 16 '11 at 14:35
  • $\begingroup$ @whuber, Thanks for the feedback. I was trying to say that $R^2$ is the most often proposed goodness-of-fit measure in regression and, if you're just starting to look into these ideas, the one that you're mostly likely to come across. I wanted to show what it is, but also why one should be cautious in using it. You're right, my answer doesn't give my suggested measure of "goodness-of-fit" because I don't think that we can tell how close we are to the true model without essentially assuming the conclusion. I do say that $R^2$ measures how close your predictions are to the sample points. $\endgroup$ – Charlie Sep 16 '11 at 14:40
  • 1
    $\begingroup$ I disagree that $R^2$ is a measure of closeness of predictions, as elaborated in my reply to the thread I linked. $\endgroup$ – whuber Sep 16 '11 at 15:10
  • $\begingroup$ I see, @whuber. Thanks for that little exposition. $\endgroup$ – Charlie Sep 16 '11 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.