4
$\begingroup$

How would one prove that the expected value of the residuals from OLS regression is zero? I will make two cases. In the first case I treat $X_i$ as random and in the second case I treat it is non-random.

First case. We know that $\hat{u}_i = y_i - \hat{y}_i$. Taking the expectation, $E[\hat{u}_i] = E[y_i] - E[\hat{y}_i]$. Now, we know from the solution of the OLS minimisation problem $\bar{y}_i = \bar{\hat{y}}_i$ because $\bar{\hat{u}} = 0$. If we take the probability limits, $plim \: \bar{y}_i = plim \: \bar{\hat{y}}_i$. By the law of large numbers this leads to $E[y_i] = E[\hat{y}_i]$. Hence, $E[\hat{u}_i] = 0$. Is this proof correct? Besides, how would one interpret $E[\hat{u}_i]$? $\hat{u}_i$ results from a given sample. Expectation is a population concept. If we take the expectation of a residual, what would this represent? The sample means of a residual term in the long run or population?

Second case. This is easy. $E[\hat{u}] = E[My] = E[M{u}] = ME[{u}] = 0$ because $My = MXB+Mu$ and $MX = 0$, because $X$ is non-random and hence can be taken out of the expectation operator, and because $E[u] = 0$. Here $M = I - P$ projection matrix. But my question is not about this case where $X$ is non-random, but the first case above where it is random.

$\endgroup$
14
  • 2
    $\begingroup$ I'm not so sure the expectations of the residuals are always zero: it ought to depend on the model. Are you talking about a standard linear multiple regression problem? As far as expectation goes, it is also a concept about random variables, and that evidently is the sense in which it is intended here. $\endgroup$
    – whuber
    Oct 1, 2018 at 19:31
  • $\begingroup$ It is the standard linear regression model. The residual is a random variable right? It depends on the random variables $y_i$ and $x_i$. It differs from one sample to another, so there is a sampling distribution for the residuals. And in fact, the interpretation of the expected value of the residuals is a question I posed. $\endgroup$
    – Snoopy
    Oct 1, 2018 at 19:47
  • 3
    $\begingroup$ Using vector notation $y=(y_1,\ldots,y_n)$ and $\hat u=(\hat u_1,\ldots, \hat u_n),$ denoting the coefficient estimates (according to the Normal equations) $$\hat\beta = (X^\prime X)^{-} X^\prime y,$$ and writing $\mathbb{I}_n$ for the $n\times n$ identity matrix, by doing nothing more than plugging everything in we obtain $$\hat u = y - \hat y = y-X\hat\beta = [\mathbb{I}_n - X (X^\prime X)^{-} X^\prime] y,$$ which explicitly is a linear transformation of $y.$ See stats.stackexchange.com/… for additional information. $\endgroup$
    – whuber
    Oct 1, 2018 at 22:05
  • 2
    $\begingroup$ 1) The mean of the residuals from OLS regression is equal to zero by construction if there is an intercept term. Given this, the expected value is zero as well - no further proof needed. 2) Why are you finding probability limits and using the Law of Large Numbers? Those are asymptotic effects, and will tell you nothing about finite-sample expected values... $\endgroup$
    – jbowman
    Oct 2, 2018 at 1:21
  • 1
    $\begingroup$ @Isabella: yes I treat the predictor as random, which was not explicit but now I made is explicit in my last edit of the question. $\endgroup$
    – Snoopy
    Oct 2, 2018 at 9:35

3 Answers 3

7
$\begingroup$

Using OLS estimation, the residuals can be written using the hat matrix $\mathbf{h} = \boldsymbol{X} (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T}$ as follows:

$$\begin{equation} \begin{aligned} \mathbf{r} &= (\mathbf{I}-\mathbf{h}) \boldsymbol{Y} \\[6pt] &= (\mathbf{I}-\mathbf{h}) (\boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt] &= (\mathbf{I}-\mathbf{h}) \boldsymbol{X} \boldsymbol{\beta} + (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon} \\[6pt] &= \mathbb{0} + (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon} \\[6pt] &= (\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}. \\[6pt] \end{aligned} \end{equation}$$

So, assuming the error terms have zero mean conditional on the explanatory variables (a standard assumption in regression analysis), you have:

$$\mathbb{E}(\mathbf{r}|\boldsymbol{X}) = \mathbb{E}((\mathbf{I}-\mathbf{h}) \boldsymbol{\varepsilon}|\boldsymbol{X}) = (\mathbf{I}-\mathbf{h}) \mathbb{E}(\boldsymbol{\varepsilon}|\boldsymbol{X}) = \mathbf{0}.$$

$\endgroup$
7
  • $\begingroup$ And by the law of iterated expectations the unconditional expected value $\textbf{r}$ is also zero, which would complete the proof? $\endgroup$
    – Snoopy
    Apr 8, 2019 at 9:16
  • 1
    $\begingroup$ If you like, but that is a weaker condition than the conditional expectation being zero. $\endgroup$
    – Ben
    Apr 8, 2019 at 9:50
  • $\begingroup$ @Ben; I argued in my answer that assumption about error term is not needed. Let me known what you think. $\endgroup$
    – markowitz
    Dec 3, 2020 at 14:00
  • 1
    $\begingroup$ @markowitz: Your answer shows that the residuals sum to zero (by construction), which is certainly an interesting property of OLS, but it is not the same as the residual vector having zero expected value. $\endgroup$
    – Ben
    Dec 3, 2020 at 21:14
  • 1
    $\begingroup$ You appear to be confusing the sample mean (which is a statistic) with the expected value of the random vector. These are two different kinds of "mean". $\endgroup$
    – Ben
    Dec 3, 2020 at 22:14
3
$\begingroup$

How would one prove that the expected value of the residuals from OLS regression is zero?

In linear regression framework many problems can emerge from the so called error term. However you here speak unambiguously about residuals in OLS context.

Then, the expected value of residuals is zero by construction. Algebra demand it; the origin come from the first order conditions of optimization for OLS parameters. Usual assumptions about error term have no role.

Following Ben's notation we can write

$$\begin{equation} \begin{aligned} \mathbf{1}' \mathbf{r} &= \mathbf{1}'(\hat{\mathbf{Y}} -\mathbf{Y}) = \mathbf{1}' \hat{\mathbf{Y}} - \mathbf{1}' \mathbf{Y} = 0 \end{aligned} \end{equation}$$

Therefore not only the expected value is zero but the sum of residuals is precisely zero too, always. ($\mathbf{1}$ is a vector of 1)

The problem of the Ben's explanation is in the second row

$$\begin{equation} \begin{aligned} \mathbf{r} &= (\mathbf{I}-\mathbf{h}) \boldsymbol{Y} \\[6pt] &= (\mathbf{I}-\mathbf{h}) (\boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt] \end{aligned} \end{equation}$$

from the decomposition of $\boldsymbol{Y}$ the assumption about the error term $\mathbb{E}(\boldsymbol{\varepsilon}|\boldsymbol{X}) = \mathbf{0}$ seems needed, but this is not. Important to note that hat matrix ($\mathbf{h}$) should be used on OLS parameters: $\mathbf{h} \boldsymbol{Y}= \boldsymbol{X}'\boldsymbol{b} = \hat{\mathbf{Y}}$

Finally we can verify that by construction: $\mathbb{E} [\boldsymbol{Y} | \boldsymbol{X} ]= \boldsymbol{X}'\boldsymbol{b} = \hat{\mathbf{Y}}$

and $\boldsymbol{Y} = \hat{\mathbf{Y}} + \mathbf{r}$

therefore $\mathbb{E} [\mathbf{r} | \boldsymbol{X} ]= \mathbb{E} [\boldsymbol{Y} | \boldsymbol{X}] - \mathbb{E} [\hat{\mathbf{Y}} | \boldsymbol{X}] = \boldsymbol{0}$

by construction too

As sidenote:

Expectation is a population concept. If we take the expectation of a residual, what would this represent? The sample means of a residual term in the long run or population?

Expectation is a general concept, you can refer it even at one observation only. For example even at just one coin tossing. The proper application depend on the context and the question.

Your context is linear regression estimated with OLS, therein the residuals are a well defined object. Important to note that residuals are an estimation quantity, you compute them. Different thing are errors, them are outside the control of researcher, them are unobservables, for this reason you have to make assumption about.

Something like "population residuals" is an ambiguous object. Residuals are all in your hands, always. You can think about these in a scheme where the amount of observations go to infinity or cover all the population. But nothing change in the above algebra and implications; them depend from the so called Geometry of OLS.

Said that, residuals remain interpretable as random variables and you can compute their expectation. Without loss of generality you can think about expectation of residuals (or estimators, ecc) as conditional of regressors ($\mathbb{E}(\mathbf{r}|\boldsymbol{X})$), so no matters if them are stocastic or not.

$\endgroup$
0
$\begingroup$

In the setting of and with the same notation as in this answer, let $\hat u_i = y_i - \hat y_i$. Then the $\hat u_i$ always sum to $0$. Indeed, by Lemma 1a and 2 of the linked answer, the hat matrix $\mathbf H$ is symmetric and has the vector $(1,1,...,1)^\top\in\mathbb R^n$ as eigenvector with eigenvalue $1$, so \begin{equation*} \sum_{i=1}^n \hat y_i = \langle (1,\dots,1)^\top, \mathbf H\, \mathbf y\rangle = \langle \mathbf H\, (1,\dots,1)^\top, \mathbf y\rangle = \langle(1,\dots,1)^\top, \mathbf y\rangle = \sum_{i=1}^n y_i. \end{equation*} It follows directly that $$\sum_{i=1}^n \hat u_i =0.$$

Note that this result is only true for linear regression with intercept. For linear regression without intercept, $(1,\dots, 1)^\top$ need not be an eigenvector of $\mathbf H$ with eigenvalue $1$, so the previous argument may fail.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.