46
$\begingroup$

I've seen "residuals" defined variously as being either "predicted minus actual values" or "actual minus predicted values". For illustration purposes, to show that both formulas are widely used, compare the following Web searches:

In practice, it almost never makes a difference, since the sign of the invidividual residuals don't usually matter (e.g. if they are squared or the absolute values are taken). However, my question is: is one of these two versions (prediction first vs. actual first) considered "standard"? I like to be consistent in my usage, so if there is a well-established conventional standard, I would prefer to follow it. However, if there is no standard, I am happy to accept that as an answer, if it can be convincingly demonstrated that there is no standard convention.

$\endgroup$
  • 8
    $\begingroup$ Since the residual is connected to the error of the model, when we write $y = a + bx + \epsilon$ it makes us think $y$ is a "fixed part" plus a "random part" so the residual is the $y$ minus the $a + bx$. $\endgroup$ – AdamO Apr 24 '18 at 14:48
  • $\begingroup$ Predicted minus actual or actual minus predicted would be prediction error (or the negative thereof), while fitted minus actual or actual minus fitted would be residual (or the negative thereof). Stephen Kolassa's answer mentions forecast errors for a reason. $\endgroup$ – Richard Hardy Apr 24 '18 at 15:54
  • $\begingroup$ I find (predicted-actual) more convenient to work with. Often you need to compute derivatives of the residual with respect to some parameters. If you use (actual-predicted), then minus signs appear that you must keep track of throughout the rest of your calculations, necessitating the use of more parentheses, making sure to cancel out double negatives when they occur, and so forth. In my experience, this leads to more errors $\endgroup$ – Nick Alger Apr 26 '18 at 15:26
42
$\begingroup$

The residuals are always actual minus predicted. The models are: $$y=f(x;\beta)+\varepsilon$$ Hence, the residuals $\hat\varepsilon$, which are estimates of errors $\varepsilon$: $$\hat\varepsilon=y-\hat y\\\hat y=f(x;\hat\beta)$$

I agree with @whuber that the sign doesn't really matter mathematically. It's just good to have a convention though. And the current convention is as in my answer.

Since OP challenged my authority on this subject, I'm adding some references:

$\endgroup$
  • 3
    $\begingroup$ I edited my question to add some sample web searches that clearly show that residuals are NOT ALWAYS actual minus predicted; the alternate is also fairly frequent--hence my confusion. My question is whether there is an authoritative documentation of the correct convention, which, unfortunately, your answer does not provide. $\endgroup$ – Tripartio Apr 24 '18 at 13:25
  • 5
    $\begingroup$ In my reading observed $-$ predicted is the majority modern convention in statistics. It's notable, however, that Gauss used the opposite convention: naturally squared residuals are the same either way in the context of least squares, sums of squares or mean squares. Although there are 19th century and earlier precedents for looking at individual residuals, caring about and particularly plotting residuals didn't start to become widespread and routine until the early 1960s. That is, it is only when the sign of the residuals is in sight that anyone needs to care what it is. $\endgroup$ – Nick Cox Apr 24 '18 at 13:35
  • 18
    $\begingroup$ +1. The concept of residual stems from "a remainder; that which is left behind": in other words, what remains in the data after the prediction has been accounted for. This suggests that whoever named these quantities a "residual" had the definition "data value minus fitted value" in mind. $\endgroup$ – whuber Apr 24 '18 at 14:06
  • 3
    $\begingroup$ @NickCox, could you please formalize your comments as an answer, with citations? My question isn't really so much about statistics as it is about scientific convention, so the kind of historical and usage insights indicated in your comment are the kind of answers I'm looking for. $\endgroup$ – Tripartio Apr 24 '18 at 14:10
  • 6
    $\begingroup$ The word residual long, long predates Salsburg. I have to say that his book, although sometimes entertaining, is far from authoritative. If interested you can seek out my review in Biometrics jstor.org/stable/3068274 $\endgroup$ – Nick Cox Apr 24 '18 at 17:21
22
$\begingroup$

I just came across a compelling reason for one answer to be the correct one.

Regression (and most statistical models of any sort) concern how the conditional distributions of a response depend on explanatory variables. An important element of the characterization of those distributions is some measure usually called "skewness" (even though various and different formulas have been offered): it refers to the most basic way in which the distributional shape departs from symmetry. Here is an example of bivariate data (a response $y$ and a single explanatory variable $x$) with positively skewed conditional responses:

![Figure 1: a scatterplot with least squares line.

The blue curve is the ordinary least squares fit. It plots the fitted values.

When we compute the difference between a response $y$ and its fitted value $\hat y$, we shift the location of the conditional distribution, but do not otherwise change its shape. In particular, its skewness will be unaltered.

Figure 2: Residuals vs. predicted values.

This is a standard diagnostic plot showing how the shifted conditional distributions vary with the predicted values. Geometrically, it's almost the same as "untilting" the previous scatterplot.

If instead we compute the difference in the other order, $\hat y - y,$ this will shift and then reverse the shape of the conditional distribution. Its skewness will be the negative of the original conditional distribution.

Figure 3: the previous plot with residuals negated

This shows the same quantities as the previous figure, but the residuals have been computed by subtracting the data from their fits--which of course is the same as negating the previous residuals.

Although both the preceding figures are mathematically equivalent in every respect--one is converted into the other simply by flipping the points across the blue horizon--one of them bears a much more direct visual relationship to the original plot.

Consequently, if our goal is to relate distributional characteristics of the residuals to the characteristics of the original data--and that almost always is the case--then it is better simply to shift the responses rather than to shift and reverse them.

The right answer is clear: compute your residuals as $y - \hat y.$

$\endgroup$
  • 1
    $\begingroup$ I don't think I follow what's special about skewness here -- doesn't your argument about the residuals matching the original plot stand by itself right away? $\endgroup$ – MichaelChirico Apr 25 '18 at 17:12
  • 2
    $\begingroup$ @Michael You are quite correct. Skewness is useful, however, for illustrating the point because it clearly distinguishes the shape of a distribution from the shape of its negative. $\endgroup$ – whuber Apr 25 '18 at 21:06
10
$\begingroup$

Green & Tashman (2008, Foresight) report on a small survey on the analogous question for forecast errors. I'll summarize arguments for either convention as reported by them:

Arguments for "actual-predicted"

  1. The statistical convention is $y=\hat{y}+\epsilon$.
  2. At least one respondent from seismology wrote that this is also the convention for modeling seismic wave traveling time. "When actual seismic wave arrives before the time predicted by model we have negative travel time residual (error)." (sic)

  3. This convention makes sense if we interpret $\hat{y}$ as a budget, plan or target. Here, a positive error means that the budget/plan/target has been exceeded.

  4. This convention makes the formulas for exponential smoothing somewhat more intuitive. We can use a $+$ sign. With the other convention, we would need to use a $-$ sign.

Arguments for "predicted-actual"

  1. If $y=\hat{y}-\epsilon$, then a positive error indicates that the forecast was too high. This is more intuitive than the converse.

    Relatedly, if a positive bias is defined as positive expected errors, it would mean that forecasts are on average too high with this convention.

    And this is pretty much the only argument given for this convention. Then again, given the misunderstandings the other convention can lead to (positive errors = forecast too low), it's a strong one.

In the end, I would argue that it comes down to who you need to communicate your residuals to. And given that there are certainly two sides to this discussion, it makes sense to explicitly note which convention you follow.

$\endgroup$
  • 7
    $\begingroup$ Interesting points, but whenever anyone says "intuitive" I translate that as "familiar to me" and the translation is often more convincing and never less. Try this: the Einstein summation convention is intuitive. Only when you've got used to it. Measuring angles from the $x$ axis counter-clockwise is intuitive. Not to geographers or anyone who learnt to use a compass before they studied coordinate geometry. $\endgroup$ – Nick Cox Apr 24 '18 at 14:52
  • 3
    $\begingroup$ @NickCox: abstractly, you are right. However, take a large number of people and ask them: "The weather forecast for today's temperature had a large positive error. Do you believe that the forecast was (A) too high or (B) too low?" I think I can predict which one of (A) or (B) an overwhelming majority will choose. $\endgroup$ – Stephan Kolassa Apr 24 '18 at 14:58
  • 6
    $\begingroup$ Yes--and if you were to phrase that question as "Do you believe the temperature was (A) higher or (B) lower than the forecast," you very well might obtain exactly the opposite answers! Referring to a "positive error" only raises the question of "what is the error," and that brings us--in a perfectly circular fashion--right back to the original question. $\endgroup$ – whuber Apr 24 '18 at 15:19
  • 2
    $\begingroup$ @whuber that's a rather unnatural phrasing of the question though. Given that the "observed" is "fixed", the relationship of the model to it seems more natural than the other way around. I get a speeding ticket for going too fast, rather than "the speed limit was below my speed". Natural language arguments definitely have a limited application to technical terms/language though/ $\endgroup$ – mbrig Apr 24 '18 at 19:19
  • 2
    $\begingroup$ @whuber What I'm saying is that one way of phrasing the question is clearly more natural (at least in English). $\endgroup$ – mbrig Apr 24 '18 at 19:30
4
$\begingroup$

Different terminology suggests different conventions. The term "residual" implies that it's what's left over after all the explanatory variables have been taken into account, i.e. actual-predicted. "Prediction error" implies that it's how much the prediction deviates from actual, i.e. prediction-actual.

One's conception of modeling also influences which convention is more natural. Suppose you have a dataframe with one or more feature columns $X = x_1,x_2...$, response column $y$, and prediction column $\hat y$.

One conception is that $y$ is the "real" value, and $\hat y$ is simply a transformed version of $X$. In this conception, $y$ and $\hat y$ are both random variables ($\hat y$ being a derived one). Although $y$ is the one we're actually interested in, $\hat y$ is the one we can observe, so $\hat y$ is used as a proxy for $y$. The "error" is how much $\hat y$ deviates from this "true" value $y$. This suggests defining the error as following the direction of this deviation, i.e. $e = \hat y -y$.

However, there's another conception that thinks of $\hat y$ as the "real" value. That is, y depends on $X$ through some deterministic process; a particular state of $X$ gives rise to a particular deterministic value. This value is then perturbed by some random process. So we have $x \rightarrow f(X)\rightarrow f(X)+error()$. In this conception, $\hat y$ is the "real" value of y. For example, suppose you're trying to calculate the value of g, the acceleration due to gravity. You drop a bunch of objects, you measure how far they fell ($X$) and how long it took them to fall ($y$). You then analyze the data with the model y = $\sqrt{\frac{2x}{g}}$. You find that there's no value of g that makes this equation work exactly. So you then model this as

$\hat y = \sqrt{\frac{2x}{g}}$
$y = \hat y +error$.

That is, you take the variable y and consider there to be a "real" value $\hat y$ that is actually being generated by physical laws, and then some other value $y$ that is $\hat y$ modified by something independent of $X$, such as measurement errors or wind gusts or whatever.

In this conception, you're taking y = $\sqrt{\frac{2x}{g}}$ to be what reality "should" be doing, and if you get answers that don't agree with that, well, reality got the wrong answer. Now of course this can seems rather silly and arrogant when put this way, but there are good reasons for proceeding this conception, and it can be useful to think this way. And ultimately, it's just a model; statisticians don't necessarily think this is actually how the world works (although there probably are some who do). And given the equation $y = \hat y +error$, it follows that errors are actual minus predicted.

Also, note that if you don't like the "reality got it wrong" aspect of the second conception, you can view it as being "We've identified some process f through which y depends on $X$, but we're not getting exactly the right answers, so there must be some other process g that's also influencing y." In this variation,

$\hat y = f(X)$
$y = \hat y+g(?)$
$g = y-\hat y$.

$\endgroup$
4
$\begingroup$

The answer by @Aksakal is completely correct, but I'll just add one additional element that I find helps me (and my students).

The motto: Statistics is "perfect". As in, I can always provide the perfect prediction (I know some eye-brows are raising right about now...so hear me out).

I'm going to predict my observed values $y_i$. With some form of model, I'll generate a predicted value for each observed value, I'll call this $\hat{y}_i$. The only problem, is that usually (always) $$y_i \ne \hat{y}_i$$ So, we'll add a new variable $\epsilon_i$ so that equality holds...but it seems to me the better option is to add it to our "predicted" ("made-up") value instead of adding it to the actual value (as adding or subtracting from an actual value may not be physically possible...see comments below): $$y_i = \hat{y}_i + \epsilon_i$$ Now, we have "perfect" prediction...our "final" value matches our observed value.

Obviously, this glosses over a tremendous amount of the statistical theory underlying what is going on...but it stress the idea that the observed value is the sum of two distinct parts (a systematic part and a random part). If you remember it in this form, you will always have that the residual, $\epsilon_i$, is the observed minus the predicted.

$\endgroup$
  • 2
    $\begingroup$ Many times, when it is written the other way, $\hat{y}_i - y_i$, it often is involved in some calculation that doesn't involve the sign (like when you are working with the absolute of the residuals or the residuals squared). $\endgroup$ – Gregg H Apr 24 '18 at 13:38
  • 6
    $\begingroup$ Why "best to add it to our predicted value"? Why not "see how much the datum needs to be adjusted to agree with our prediction"? Neither approach seems to have a claim of being more obvious, meaningful, or "intuitive" than the other. $\endgroup$ – whuber Apr 24 '18 at 14:07
  • 2
    $\begingroup$ @whuber one item is "real" (observed, concrete), the other is a (hypothetical) construct; if we were modeling height based on weight, ¿would it be reasonable to "shrink" someone by 3 inches just to match their actual/observed height to some (imaginary) predicted value? $\endgroup$ – Gregg H Apr 24 '18 at 16:09
  • 2
    $\begingroup$ Yes--that's a common way of thinking about data. I'm only trying to point out the possibility that your assumptions about how people will perceive this question and understand the meaning of "best" might be speculative and subjective. $\endgroup$ – whuber Apr 24 '18 at 16:25
  • $\begingroup$ fair point...will update with brief comment $\endgroup$ – Gregg H Apr 24 '18 at 16:33
2
$\begingroup$

$\newcommand{\e}{\varepsilon}$I'm going to use the particular case of least squares linear regression. If we take our model to be $Y = X\beta + \e$ then as @Aksakal points out we naturally end up with $\e = Y - X\beta$ so $\hat \e = Y - \hat Y$. If instead we take $Y = X\beta - \e$ as our model, which we are certainly free to do, then we get $\e = X\beta - Y \implies \hat \e = \hat Y - Y$. At this point there's really no reason to prefer one over the other aside from a vague preference for $1$ over $-1$.

But if $\hat \e = Y - \hat Y$ then we obtain our residuals via $(I - P_X)Y$, where $I - P_X$ is an idempotent matrix projecting into the space orthogonal to the column space of the design matrix $X$. If we instead used $Y = X\beta - \e$ then we end up with $\hat \e = (P_X - I)Y$. But $P_X - I$ is not itself idempotent as $(P_X - I)^2 = P_X^2 - 2P_X + I = -(P_X - I)$. So really $P_X - I$ is the negative of a projection matrix, namely $I - P_X$. So I view this as undoing the negative introduced by using $Y = X\beta - \e$, so for the sake of parsimony it's better to just use $Y = X\beta + \e$ which in turn gives us $Y - \hat Y$ as the residuals.

As mentioned elsewhere it's not like anything breaks if we use $\hat Y - Y$, but we end up with this double negative situation which I think is a good enough reason to just use $Y - \hat Y$.

$\endgroup$
  • $\begingroup$ But writing anything $+ e$ has nothing to do with the signs of particular values of $e$, any more than writing $y = \beta_0 + \beta_1 x$ is a commitment or assumption that $\beta_0$ or $\beta_1$ is positive in practice. It could be the same equation but with $e$ reversed in sign. $\endgroup$ – Nick Cox Apr 24 '18 at 14:47
  • $\begingroup$ @NickCox thank you for your comment, I realize I had predicated my answer on the assumption that we'd want to write our model $Y = X\beta + \varepsilon$. I've rewritten it to address this $\endgroup$ – jld Apr 24 '18 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.