7
$\begingroup$

I asked this question and thought I found a solution to it by working through the steps in my textbook. Unfortunately the comments told me that I was completely wrong and only got the proof to work by chance, so now I'm really lost and frustrated.

To fix notation, the model is $$ y_i = \beta_0 + \beta_1 x_i + u_i, $$ where $u_i$ is the error term.

Earlier I showed that $\hat{\beta}_1 = \beta_1 + \sum_{i=1}^n w_i u_i$, where $w_i = \frac{x_i - \bar{x}}{SST_x}$, and $SST_x = \sum_{i=1}^n (x_i - \bar{x})^2$.

Here is what I did so far: \begin{align} E[(\hat{\beta_1}-\beta_1) \bar{u}] &= E[\bar{u}\displaystyle\sum\limits_{i=1}^n w_i u_i] \\ &=\displaystyle\sum\limits_{i=1}^n E[w_i \bar{u} u_i] \\ &=\displaystyle\sum\limits_{i=1}^n w_i E[\bar{u} u_i] \\ &= \displaystyle\sum\limits_{i=1}^n w_i \left(Cov(\bar{u}, u_i) + E(\bar{u})E(u_i)\right) \\ &= \displaystyle\sum\limits_{i=1}^n w_i Cov(\bar{u}, u_i) \\ &= \displaystyle\sum\limits_{i=1}^n w_i Cov\left(\frac{1}{n} \displaystyle\sum\limits_{i=1}^n u_i, u_i\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i Cov\left(\displaystyle\sum\limits_{i=1}^n u_i, u_i\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(\left[u_i - E(u_i)\right]\left[ \displaystyle\sum\limits_{i=1}^n u_i - E\left(\displaystyle\sum\limits_{i=1}^n u_i\right)\right]\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(\left[u_i - E(u_i)\right]\left[ \displaystyle\sum\limits_{i=1}^n u_i - \displaystyle\sum\limits_{i=1}^n E\left(u_i\right)\right]\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(\left[u_i - 0\right]\left[ \displaystyle\sum\limits_{i=1}^n u_i - 0\right]\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(u_i\displaystyle\sum\limits_{i=1}^n u_i\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E\left(u_i u_1 +\cdots + u_i u_n \right)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E\left(u_i u_1\right) +\cdots + E\left(u_i u_n \right)\right] \\ \end{align}

At this point, I want to say that because we assume the errors for each observation are i.i.d:

\begin{align} &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E(u_i) E(u_1) +\cdots + E(u_i) E(u_n)\right] \\ \end{align}

But if this is the case then since $E(u_i) = 0, \forall i$, everything just cancels and I don't need to use the definition of $w_i$, which one of the comments mentioned I needed to use last time.

This is in chapter 2 of my book and the book says that "it's obvious that..." so maybe it is obvious. I keep asking for hints but I feel like I don't understand them, but I want to keep trying.

Is this correct? If not, does anyone have a hint on this part of the problem, assuming I'm on the right path? I can probably make the steps less verbose, but I want to make sure it's right first. The last time I worked this out, I reached the right conclusion but the proof was all wrong, so I'm assuming I only have it right by chance again.

$\endgroup$
  • 1
    $\begingroup$ Two things on the proposed proof: (a) on the second line, the factor $1/n$ shouldn't appear as ${\rm E}(\sum R_i) = \sum{\rm E}( R_i)$; (b) on line 8, there is an expectation missing since ${\rm cov}(R, S) = {\rm E} \left[\left\{ R - {\rm E}(R) \right\}\left\{ S - {\rm E}(S) \right\}\right]$. $\endgroup$ – QuantIbex Jul 17 '13 at 23:07
  • $\begingroup$ @QuantIbex Dumb typos, thank you. I'm still not using the definition of $w_i$ since I just get everything to cancel, so this can't be right. $\endgroup$ – M T Jul 18 '13 at 3:26
  • 1
    $\begingroup$ Hint: the relation ${\rm E}(u_i u_j) = {\rm E}(u_i){\rm E}(u_j)$ holds for all $j$ except one. Which one is it? $\endgroup$ – QuantIbex Jul 18 '13 at 9:22
  • $\begingroup$ @QuantIbex Right! So we have \begin{align} &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E\left(u_i u_1\right) +\cdots + E(u_i u_i) + \cdots+ E\left(u_i u_n \right)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E(u_i) E(u_1) +\cdots + E(u_i^2) + \cdots + E(u_i) E(u_n)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E(u_i^2) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[Var(u_i) + E(u_i) E(u_i)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \sigma^2 \\ \end{align} $\endgroup$ – M T Jul 18 '13 at 16:38
  • 1
    $\begingroup$ Why can the expectations operator move past w? $\endgroup$ – beets Oct 11 '15 at 22:17
4
$\begingroup$

Strategy

It can be illuminating to move back and forth among three points of view: the statistical one (viewing $x_i$ and $y_i$ as data), a geometrical one (where least squares solutions are just projections in suitable Euclidean spaces), and an algebraic one (manipulating symbols representing matrices or linear transformations). Doing this not only streamlines the ideas, but also exposes the assumptions needed to make the result true, which otherwise might be buried in all the summations.

Notation and assumptions

So: let $y = X\beta + u$ where $y$ is an $n$-vector, $X$ is an $n$ by $p+1$ "design matrix" whose first column is all ones, $\beta$ is a $p+1$-vector of true coefficients, and $u$ are iid random variables with zero expectation and common variance $\sigma^2$. (Let's continue, as in the question, to begin the coefficient vector's indexes with $0$, writing ${\beta} = ({\beta_0}, {\beta_1}, \ldots, {\beta_p})$.) This generalizes the question, for which $X$ has just two columns: its second column is the vector $(x_1, x_2, \ldots, x_n)'$.

Basic properties of OLS regression

The regression estimate $\hat{\beta}$ is a $p+1$-vector obtained by applying a linear transformation $\mathbb{P}$ to $y$. As the solution to the regression, it projects the exact values $X\beta$ onto the true values $\beta$: $$\mathbb{P}\left(X\beta\right) = \beta.$$ Finally--and this is the crux of the matter before us--it is obvious statistically (thinking of these values as data)--that the projection of $y = (1, 1, \ldots, 1)'$ has the unique solution $\hat{\beta} = (1, 0, 0, \ldots, 0)$: $$\mathbb{P}1_n' = (1, 0, 0, \ldots, 0),$$ because when all the responses $y_i$ are equal to $1$, the intercept $\beta_0 = 1$ and all the other coefficients must vanish. That's all we need to know. (Having a formula for $\mathbb{P}$ in terms of $X$ is unimportant (and distracting).)

Easy preliminaries

Begin with some straightforward algebraic manipulation of the original expression:

$$\eqalign{ (\hat{\beta}-\beta)\bar{u} &= (\mathbb{P}y-\beta)\bar{u} \\ &= (\mathbb{P}(X\beta+u)-\beta)\bar{u} \\ &= \mathbb{P}(X\beta+u)\bar{u} - \beta\bar{u} \\ &= (\mathbb{P}X\beta)\bar{u} + \mathbb{P}u\bar{u} - \beta\bar{u} \\ &= \beta\bar{u} + \mathbb{P}u\bar{u} - \beta\bar{u}\\ &= \mathbb{P}(u\bar{u}). } $$

This almost mindless sequence of steps--each leads naturally to the next by simple algebraic rules--is motivated by the desire to (a) express the random variation purely in terms of $u$, whence it all derives, and (b) introduce $\mathbb{P}$ so that we can exploit its properties.

Computing the expectation

Taking the expectation can no longer be put off, but because $\mathbb{P}$ is a linear operator, it will be applied to the expectation of $u\bar{u}$. We could employ some formal matrix operations to work out this expectation, but there's an easier way. Recalling that the $u_i$ are iid, it is immediate that all the coefficients in $\mathbb{E}[u\bar{u}]$ must be the same. Since they are the same, each one equals their average. This can be obtained by averaging the coefficients of $u$, multiplying by $\bar{u}$, and taking the expectation. But that's just a recipe for finding $$\mathbb{E}[\bar{u}\bar{u}] = \text{Var}[\bar{u}] = \sigma^2/n.$$ It follows that $\mathbb{E}[u\bar{u}]$ is a vector of $n$ values, all of which equal $\sigma^2/n$. Using our previous vector shorthand we may write $$\mathbb{E}[(\hat{\beta}-\beta)\bar{u} ] = \mathbb{E}[\mathbb{P}(u\bar{u})] = \mathbb{PE}[u\bar{u}]=\mathbb{P}1_n'\sigma^2/n=(\sigma^2/n, 0, 0, \ldots, 0).$$

Conclusion

This says that the estimated coefficients $\hat{\beta_i}$ and the mean error $\hat{u}$ are uncorrelated for $i=1, 2, \ldots, p$, but not so for $\hat{\beta_0}$ (the intercept).


It is instructive to review the steps and consider which assumptions were essential and which elements of the regression apparatus simply did not appear in the demonstration. We should expect any proof, no matter how elementary or sophisticated, will need to use the same (or stronger) assumptions and will need, in some guise or another, to include calculations of $\mathbb{P}1_n'$ and $\mathbb{E}[u\bar{u}]$.

$\endgroup$
  • $\begingroup$ This is really helpful. One clarification: "Recalling that the $u_i$ are iid, it is immediate that all the coefficients in $E[u\bar{u}]$ must be the same. Since they are the same, each one equals their average. This can be obtained by averaging the coefficients of $u$, multiplying by $\bar{u}$, and taking the expectation." What coefficients are you referring to? That part still confuses me a little. $\endgroup$ – M T Jul 18 '13 at 17:30
  • $\begingroup$ $u$ is a vector with $n$ coefficients, $u_1, u_2, \ldots, u_n$. Thus, $\mathbb{E}[u\bar{u}]$, being the expectation of a vector $u$ times a scalar $\bar{u}$, itself is a vector with $n$ coefficients $\mathbb{E}[u_i\bar{u}]$. The parallels with the working-out in your comments to the question should be clear. What is illuminating about this symmetry-based argument is that it shows how the iid assumption implies the conclusion. $\endgroup$ – whuber Jul 18 '13 at 17:34
  • $\begingroup$ Just so I'm clear. All the coeffs in $E[u\bar{u}]$ are the same because \begin{align} E[u \bar{u}] &= E \begin{bmatrix} u_1 \bar{u} \\ \vdots \\ u_n \bar{u} \end{bmatrix} \\ &= \begin{bmatrix} E[u_1 \bar{u}] \\ \vdots \\ E[u_n \bar{u}] \end{bmatrix} \\ &= \frac{1}{n} \begin{bmatrix} E[u_1 \sum\limits_{j=1}^n u_i] \\ \vdots \\ E[u_n \sum\limits_{j=1}^n u_i] \end{bmatrix} \\ &=\frac{1}{n} \begin{bmatrix} E[u_1 u_1] + \cdots + E[u_1 u_n] \\ \vdots \\ E[u_n u_1] + \cdots + E[u_n u_n] \\ \end{bmatrix} \\ &=\frac{1}{n} \begin{bmatrix} \sigma^2 \\ \vdots \\ \sigma^2 \\ \end{bmatrix} \\ \end{align} $\endgroup$ – M T Jul 18 '13 at 20:31
  • $\begingroup$ You are correct, but that is not the reasoning I presented here: I am intentionally avoiding that calculation. After the second equality you only have to observe that $\mathbb{E}[u_i\bar{u}] = \mathbb{E}[u_j\bar{u}]$ for all $i$ and $j$ because the $u_i$ are iid (actually, exchangeability would suffice) and you can jump directly to the conclusion. There's no need for all the extra algebra. $\endgroup$ – whuber Jul 18 '13 at 20:41
  • 1
    $\begingroup$ Right, the $u_i$ are exchangeable, so once you know one $e[u_i \bar{u}]$, you know all of them and you've found the entire vector. Thank you for the great answer. $\endgroup$ – M T Jul 19 '13 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.