# Consistency of unbiased estimator of error term variance in Multiple regression

Let $Y=X\beta+\epsilon$. We know that $\frac{e'e}{n-k}$ is an unbiased estimator of $Var(\epsilon)$, where $e$ is the vector of residuals, and $\epsilon$ is multivariate normal distributed in this model.

How can we prove, if it is, that this estimator converges to the right value? Or even if it's consistent?

Any help would be appreciated.

You know that $S^2_e = \frac{1}{n-k}\mathbf{e}'\mathbf{e}$ is an unbiased estimator of $\sigma^2$. So, if you show that $Var(S^2_e) \to 0$ as $n \to \infty$ you've shown that $S^2_e$ is consistent (i.e., converges to the right value).

This isn't difficult: $\mathbf{e}'\mathbf{e} \sim \sigma^2\cdot\chi^2_{n-k}$. Consequently, the variance of $\mathbf{e}'\mathbf{e}$ is $2(n-k)\sigma^4$. The variance of $S^2_e$ then is $\frac{1}{(n-k)^2}\cdot Var(\mathbf{e}'\mathbf{e}) = \frac{2\sigma^4}{n-k}$, which obviously tends to $0$ as $n$ grows without bound.

• One doubt: Why $\mathbf{e'e} \sim \sigma^2 \chi^2_{n-k}$? Because from my calculations $e_i=(X(\beta-b))_i+\epsilon_i \sim N((X(\beta-b))_i, \sigma^2)$ To be chi-squared the mean shouldn't be zero? – An old man in the sea. Aug 9 '14 at 20:43
• *edit from comment above: To be $\chi^2$ shouldn't we have a sum of squares of standard normals? – An old man in the sea. Aug 9 '14 at 20:52
• The genesis of the $\chi^2$ distribution is as the square of a standard normal. That is far from the only place it arises, however. There is a hint about what is going on: $\mathbf{e}$ has $n$ elements, so $\mathbf{e}'\mathbf{e}$ is a sum of $n$ terms. But it has only $n-k$ degrees of freedom. For some of the gory details, see Cochran's Theorem in a Linear Models text (I prefer Graybill, but there are lots of alternatives) or on Wikipedia (en.wikipedia.org/wiki/Cochran's_theorem). – Dennis Aug 9 '14 at 20:58
• @an-old-man-in-the-sea ...and this is why the sum of squared residuals equals a scaled chi-square (i.e. a gamma distribution): because the sum of squared residuals divided by their variance follows a chi-square proper. As for the degrees of freedom, it's indeed Cochran's Theorem. – Alecos Papadopoulos Aug 9 '14 at 21:29
• an-old-man-in-the-sea As for linear models, it depends on your level of course but Dobson's book is a good introduction, books.google.gr/books/about/… Still, I have the impression that @Dennis could provide a much more authoritative suggestion. – Alecos Papadopoulos Aug 9 '14 at 21:36

The residuals can be expressed in terms of the true error terms as

$$e_i = \epsilon_i - \mathbf x_i'(\hat \beta - \beta)$$

Squaring,

$$e_i^2 = [\epsilon_i - \mathbf x_i'(\hat \beta - \beta)]^2 = \epsilon_i^2 -2 (\hat \beta - \beta)'\mathbf x_i\epsilon_i + (\hat \beta - \beta)'\mathbf x_i\mathbf x_i'(\hat \beta - \beta)$$

Then

$$\frac{\mathbf e'\mathbf e}{n-K}=\frac {1}{n-K}\sum_{i=1}^n e_i^2 = \frac {1}{n-K}\sum_{i=1}^n \epsilon_i^2 -\frac {2}{n-K}\sum_{i=1}^n (\hat \beta - \beta)'\mathbf x_i\epsilon_i \\ + \frac {1}{n-K}\sum_{i=1}^n(\hat \beta - \beta)'\mathbf x_i\mathbf x_i'(\hat \beta - \beta)$$

Note that $$(\hat \beta - \beta)$$ does not depend on the sum index so

$$\frac {1}{n-K}\sum_{i=1}^n e_i^2 = \frac {1}{n-K}\sum_{i=1}^n \epsilon_i^2 -(\hat \beta - \beta)'\frac {2}{n-K}\sum_{i=1}^n \mathbf x_i\epsilon_i + (\hat \beta - \beta)'\left[\frac {1}{n-K}\sum_{i=1}^n\mathbf x_i\mathbf x_i'\right](\hat \beta - \beta)$$

Considering the limit $$n \to \infty$$,

1. Under the assumptions of ergodicity, that the 2nd moment of the error term exists, and that $$E(\epsilon) = 0$$ we have by Kinchin's LLN

$$\frac {1}{n-K}\sum_{i=1}^n e_i^2 \to_p E(\epsilon^2) = \text{Var}(\epsilon)$$

1. Under the assumption $$E(\mathbf x_i\epsilon_i)=\mathbf 0$$ and given ergodicity,

$$(\hat \beta - \beta)' \to_p 0, \;\;\;\frac {2}{n-K}\sum_{i=1}^n \mathbf x_i\epsilon_i \to_p 2E(\mathbf x_i\epsilon_i)=0$$ so the second term goes to zero in probability

1. Given the Grenander conditions on the regressor matrix, and ergodicity,

$$\left[\frac {1}{n-K}\sum_{i=1}^n\mathbf x_i\mathbf x_i'\right] \to_p \mathbf Q < \infty$$

and since $$(\hat \beta - \beta)' \to_p 0$$, the the third term also goes to zero in probability.

So we are left only with the first term, and we have consistency

• +1 It is worth pointing out that this proof does not require the normality assumption. – Christoph Hanck Mar 22 at 19:14
• @ChristophHanck Good point, thanks for making it. – Alecos Papadopoulos Mar 22 at 19:16