# Consistency of unbiased estimator of error term variance in Multiple regression

Let $Y=X\beta+\epsilon$. We know that $\frac{e'e}{n-k}$ is an unbiased estimator of $Var(\epsilon)$, where $e$ is the vector of residuals, and $\epsilon$ is multivariate normal distributed in this model.

How can we prove, if it is, that this estimator converges to the right value? Or even if it's consistent?

Any help would be appreciated.

You know that $S^2_e = \frac{1}{n-k}\mathbf{e}'\mathbf{e}$ is an unbiased estimator of $\sigma^2$. So, if you show that $Var(S^2_e) \to 0$ as $n \to \infty$ you've shown that $S^2_e$ is consistent (i.e., converges to the right value).

This isn't difficult: $\mathbf{e}'\mathbf{e} \sim \sigma^2\cdot\chi^2_{n-k}$. Consequently, the variance of $\mathbf{e}'\mathbf{e}$ is $2(n-k)\sigma^4$. The variance of $S^2_e$ then is $\frac{1}{(n-k)^2}\cdot Var(\mathbf{e}'\mathbf{e}) = \frac{2\sigma^4}{n-k}$, which obviously tends to $0$ as $n$ grows without bound.

• One doubt: Why $\mathbf{e'e} \sim \sigma^2 \chi^2_{n-k}$? Because from my calculations $e_i=(X(\beta-b))_i+\epsilon_i \sim N((X(\beta-b))_i, \sigma^2)$ To be chi-squared the mean shouldn't be zero? Commented Aug 9, 2014 at 20:43
• *edit from comment above: To be $\chi^2$ shouldn't we have a sum of squares of standard normals? Commented Aug 9, 2014 at 20:52
• The genesis of the $\chi^2$ distribution is as the square of a standard normal. That is far from the only place it arises, however. There is a hint about what is going on: $\mathbf{e}$ has $n$ elements, so $\mathbf{e}'\mathbf{e}$ is a sum of $n$ terms. But it has only $n-k$ degrees of freedom. For some of the gory details, see Cochran's Theorem in a Linear Models text (I prefer Graybill, but there are lots of alternatives) or on Wikipedia (en.wikipedia.org/wiki/Cochran's_theorem). Commented Aug 9, 2014 at 20:58
• @an-old-man-in-the-sea ...and this is why the sum of squared residuals equals a scaled chi-square (i.e. a gamma distribution): because the sum of squared residuals divided by their variance follows a chi-square proper. As for the degrees of freedom, it's indeed Cochran's Theorem. Commented Aug 9, 2014 at 21:29
• an-old-man-in-the-sea As for linear models, it depends on your level of course but Dobson's book is a good introduction, books.google.gr/books/about/… Still, I have the impression that @Dennis could provide a much more authoritative suggestion. Commented Aug 9, 2014 at 21:36

The residuals can be expressed in terms of the true error terms as

$$e_i = \epsilon_i - \mathbf x_i'(\hat \beta - \beta)$$

Squaring,

$$e_i^2 = [\epsilon_i - \mathbf x_i'(\hat \beta - \beta)]^2 = \epsilon_i^2 -2 (\hat \beta - \beta)'\mathbf x_i\epsilon_i + (\hat \beta - \beta)'\mathbf x_i\mathbf x_i'(\hat \beta - \beta)$$

Then

$$\frac{\mathbf e'\mathbf e}{n-K}=\frac {1}{n-K}\sum_{i=1}^n e_i^2 = \frac {1}{n-K}\sum_{i=1}^n \epsilon_i^2 -\frac {2}{n-K}\sum_{i=1}^n (\hat \beta - \beta)'\mathbf x_i\epsilon_i \\ + \frac {1}{n-K}\sum_{i=1}^n(\hat \beta - \beta)'\mathbf x_i\mathbf x_i'(\hat \beta - \beta)$$

Note that $$(\hat \beta - \beta)$$ does not depend on the sum index so

$$\frac {1}{n-K}\sum_{i=1}^n e_i^2 = \frac {1}{n-K}\sum_{i=1}^n \epsilon_i^2 -(\hat \beta - \beta)'\frac {2}{n-K}\sum_{i=1}^n \mathbf x_i\epsilon_i + (\hat \beta - \beta)'\left[\frac {1}{n-K}\sum_{i=1}^n\mathbf x_i\mathbf x_i'\right](\hat \beta - \beta)$$

Considering the limit $$n \to \infty$$,

1. Under the assumptions of ergodicity, that the 2nd moment of the error term exists, and that $$E(\epsilon) = 0$$ we have by Kinchin's LLN

$$\frac {1}{n-K}\sum_{i=1}^n e_i^2 \to_p E(\epsilon^2) = \text{Var}(\epsilon)$$

1. Under the assumption $$E(\mathbf x_i\epsilon_i)=\mathbf 0$$ and given ergodicity,

$$(\hat \beta - \beta)' \to_p 0, \;\;\;\frac {2}{n-K}\sum_{i=1}^n \mathbf x_i\epsilon_i \to_p 2E(\mathbf x_i\epsilon_i)=0$$ so the second term goes to zero in probability

1. Given the Grenander conditions on the regressor matrix, and ergodicity,

$$\left[\frac {1}{n-K}\sum_{i=1}^n\mathbf x_i\mathbf x_i'\right] \to_p \mathbf Q < \infty$$

and since $$(\hat \beta - \beta)' \to_p 0$$, the the third term also goes to zero in probability.

So we are left only with the first term, and we have consistency

• +1 It is worth pointing out that this proof does not require the normality assumption. Commented Mar 22, 2019 at 19:14
• @ChristophHanck Good point, thanks for making it. Commented Mar 22, 2019 at 19:16