Are residuals "predicted minus actual" or "actual minus predicted"

Question

I've seen "residuals" defined variously as being either "predicted minus actual values" or "actual minus predicted values". For illustration purposes, to show that both formulas are widely used, compare the following Web searches:

• residual "predicted minus actual"
• residual "actual minus predicted"

In practice, it almost never makes a difference, since the sign of the invidividual residuals don't usually matter (e.g. if they are squared or the absolute values are taken). However, my question is: is one of these two versions (prediction first vs. actual first) considered "standard"? I like to be consistent in my usage, so if there is a well-established conventional standard, I would prefer to follow it. However, if there is no standard, I am happy to accept that as an answer, if it can be convincingly demonstrated that there is no standard convention.

Answer 1

The residuals are always actual minus predicted. The models are: $$y=f(x;\beta)+\varepsilon$$ Hence, the residuals $\hat\varepsilon$, which are estimates of errors $\varepsilon$: $$\hat\varepsilon=y-\hat y\\\hat y=f(x;\hat\beta)$$

I agree with @whuber that the sign doesn't really matter mathematically. It's just good to have a convention though. And the current convention is as in my answer.

Since OP challenged my authority on this subject, I'm adding some references:

• "(2008) Residual. In: The Concise Encyclopedia of Statistics. Springer, New York, NY, which gives the same definition.
• Fisher's "Statistical Methods for Research Workers" 1925, has the same definition too, see Section 26 in this 1934 version. Despite unassuming title, this is an important work in historical context

Answer 2

I just came across a compelling reason for one answer to be the correct one.

Regression (and most statistical models of any sort) concern how the conditional distributions of a response depend on explanatory variables. An important element of the characterization of those distributions is some measure usually called "skewness" (even though various and different formulas have been offered): it refers to the most basic way in which the distributional shape departs from symmetry. Here is an example of bivariate data (a response $y$ and a single explanatory variable $x$) with positively skewed conditional responses:

The blue curve is the ordinary least squares fit. It plots the fitted values.

When we compute the difference between a response $y$ and its fitted value $\hat y$, we shift the location of the conditional distribution, but do not otherwise change its shape. In particular, its skewness will be unaltered.

This is a standard diagnostic plot showing how the shifted conditional distributions vary with the predicted values. Geometrically, it's almost the same as "untilting" the previous scatterplot.

If instead we compute the difference in the other order, $\hat y - y,$ this will shift and then reverse the shape of the conditional distribution. Its skewness will be the negative of the original conditional distribution.

This shows the same quantities as the previous figure, but the residuals have been computed by subtracting the data from their fits--which of course is the same as negating the previous residuals.

Although both the preceding figures are mathematically equivalent in every respect--one is converted into the other simply by flipping the points across the blue horizon--one of them bears a much more direct visual relationship to the original plot.

Consequently, if our goal is to relate distributional characteristics of the residuals to the characteristics of the original data--and that almost always is the case--then it is better simply to shift the responses rather than to shift and reverse them.

The right answer is clear: compute your residuals as $y - \hat y.$

Answer 3

Green & Tashman (2008, Foresight) report on a small survey on the analogous question for forecast errors. I'll summarize arguments for either convention as reported by them:

Arguments for "actual-predicted"

1. The statistical convention is $y=\hat{y}+\epsilon$.

2. At least one respondent from seismology wrote that this is also the convention for modeling seismic wave traveling time. "When actual seismic wave arrives before the time predicted by model we have negative travel time residual (error)." (sic)

3. This convention makes sense if we interpret $\hat{y}$ as a budget, plan or target. Here, a positive error means that the budget/plan/target has been exceeded.

4. This convention makes the formulas for exponential smoothing somewhat more intuitive. We can use a $+$ sign. With the other convention, we would need to use a $-$ sign.

Arguments for "predicted-actual"

1. If $y=\hat{y}-\epsilon$, then a positive error indicates that the forecast was too high. This is more intuitive than the converse.

   Relatedly, if a positive bias is defined as positive expected errors, it would mean that forecasts are on average too high with this convention.

   And this is pretty much the only argument given for this convention. Then again, given the misunderstandings the other convention can lead to (positive errors = forecast too low), it's a strong one.

In the end, I would argue that it comes down to who you need to communicate your residuals to. And given that there are certainly two sides to this discussion, it makes sense to explicitly note which convention you follow.

Answer 4

Different terminology suggests different conventions. The term "residual" implies that it's what's left over after all the explanatory variables have been taken into account, i.e. actual-predicted. "Prediction error" implies that it's how much the prediction deviates from actual, i.e. prediction-actual.

One's conception of modeling also influences which convention is more natural. Suppose you have a dataframe with one or more feature columns $X = x_1,x_2...$, response column $y$, and prediction column $\hat y$.

One conception is that $y$ is the "real" value, and $\hat y$ is simply a transformed version of $X$. In this conception, $y$ and $\hat y$ are both random variables ($\hat y$ being a derived one). Although $y$ is the one we're actually interested in, $\hat y$ is the one we can observe, so $\hat y$ is used as a proxy for $y$. The "error" is how much $\hat y$ deviates from this "true" value $y$. This suggests defining the error as following the direction of this deviation, i.e. $e = \hat y -y$.

However, there's another conception that thinks of $\hat y$ as the "real" value. That is, y depends on $X$ through some deterministic process; a particular state of $X$ gives rise to a particular deterministic value. This value is then perturbed by some random process. So we have $x \rightarrow f(X)\rightarrow f(X)+error()$. In this conception, $\hat y$ is the "real" value of y. For example, suppose you're trying to calculate the value of g, the acceleration due to gravity. You drop a bunch of objects, you measure how far they fell ($X$) and how long it took them to fall ($y$). You then analyze the data with the model y = $\sqrt{\frac{2x}{g}}$. You find that there's no value of g that makes this equation work exactly. So you then model this as

$\hat y = \sqrt{\frac{2x}{g}}$
$y = \hat y +error$.

That is, you take the variable y and consider there to be a "real" value $\hat y$ that is actually being generated by physical laws, and then some other value $y$ that is $\hat y$ modified by something independent of $X$, such as measurement errors or wind gusts or whatever.

In this conception, you're taking y = $\sqrt{\frac{2x}{g}}$ to be what reality "should" be doing, and if you get answers that don't agree with that, well, reality got the wrong answer. Now of course this can seems rather silly and arrogant when put this way, but there are good reasons for proceeding this conception, and it can be useful to think this way. And ultimately, it's just a model; statisticians don't necessarily think this is actually how the world works (although there probably are some who do). And given the equation $y = \hat y +error$, it follows that errors are actual minus predicted.

Also, note that if you don't like the "reality got it wrong" aspect of the second conception, you can view it as being "We've identified some process f through which y depends on $X$, but we're not getting exactly the right answers, so there must be some other process g that's also influencing y." In this variation,

$\hat y = f(X)$
$y = \hat y+g(?)$
$g = y-\hat y$.

Answer 5

The answer by @Aksakal is completely correct, but I'll just add one additional element that I find helps me (and my students).

The motto: Statistics is "perfect". As in, I can always provide the perfect prediction (I know some eye-brows are raising right about now...so hear me out).

I'm going to predict my observed values $y_i$. With some form of model, I'll generate a predicted value for each observed value, I'll call this $\hat{y}_i$. The only problem, is that usually (always) $$y_i \ne \hat{y}_i$$ So, we'll add a new variable $\epsilon_i$ so that equality holds...but it seems to me the better option is to add it to our "predicted" ("made-up") value instead of adding it to the actual value (as adding or subtracting from an actual value may not be physically possible...see comments below): $$y_i = \hat{y}_i + \epsilon_i$$ Now, we have "perfect" prediction...our "final" value matches our observed value.

Obviously, this glosses over a tremendous amount of the statistical theory underlying what is going on...but it stress the idea that the observed value is the sum of two distinct parts (a systematic part and a random part). If you remember it in this form, you will always have that the residual, $\epsilon_i$, is the observed minus the predicted.

Answer 6

$\newcommand{\e}{\varepsilon}$I'm going to use the particular case of least squares linear regression. If we take our model to be $Y = X\beta + \e$ then as @Aksakal points out we naturally end up with $\e = Y - X\beta$ so $\hat \e = Y - \hat Y$. If instead we take $Y = X\beta - \e$ as our model, which we are certainly free to do, then we get $\e = X\beta - Y \implies \hat \e = \hat Y - Y$. At this point there's really no reason to prefer one over the other aside from a vague preference for $1$ over $-1$.

But if $\hat \e = Y - \hat Y$ then we obtain our residuals via $(I - P_X)Y$, where $I - P_X$ is an idempotent matrix projecting into the space orthogonal to the column space of the design matrix $X$. If we instead used $Y = X\beta - \e$ then we end up with $\hat \e = (P_X - I)Y$. But $P_X - I$ is not itself idempotent as $(P_X - I)^2 = P_X^2 - 2P_X + I = -(P_X - I)$. So really $P_X - I$ is the negative of a projection matrix, namely $I - P_X$. So I view this as undoing the negative introduced by using $Y = X\beta - \e$, so for the sake of parsimony it's better to just use $Y = X\beta + \e$ which in turn gives us $Y - \hat Y$ as the residuals.

As mentioned elsewhere it's not like anything breaks if we use $\hat Y - Y$, but we end up with this double negative situation which I think is a good enough reason to just use $Y - \hat Y$.

Answer 7

For practical purposes it's better to calculate residuals as "actual minus predicted": When trying to guess what the physical cause of an outlier data point might be, it gives a better intuitive sense to have an unusually large positive observed value have a large positive residual, and this is precisely what you get when you calculate the residual in the usual way as "actual minus predicted". See also Stephan Kolassa's summary above of four arguments for "actual minus predicted".

The textbooks from which I learnt always taught residuals were (Observed - Expected), in line with Fisher historically as noted by @Aksakal, and contrary to Gauss as noted by @NickCox. Philosophically this corresponds to assuming our model is perfect, and observed deviations from our model are random errors. Although the reverse (predicted minus actual) is equivalent for mathematical purposes, in philosophical terms it would be assuming the data is perfect and the model inadequate...which is true but not helpful and might lead to boring discussions over the scientific method, statistical testing and Bayesian thinking.

In terms of popularity, references to "Observed minus Expected" residuals are listed more than twelve times as often as the reverse in a Google search today, so O-E ("actual minus predicted") is clearly still the "standard".