OLS Regression : Efficiency of the estimator of the variance of the residuals under the assumption of normality

Question

My question is probably already answered somewhere but I did not find it.

In the standard linear regression model under the assumption that residuals are normally distributed, we have a result stronger than the Gauss-Markov theorem : $\hat{\beta}$ is efficient, that is to say has the smalles variance among all unbiased estimators. It can be shown by showing its variance is the one given by the Cramer-Rao bound.

But what can we say about the estimator of the variance of the residuals ? We know that it is

$$ \left(\hat{\sigma}^{OLS}_u\right)^2 = \frac{\hat{u}^T\hat{u}}{T-(k+1)} $$

And therefore $$ \mathbb{V}\left[\left(\hat{\sigma}^{OLS}_u\right)^2\right]=\frac{2\sigma^4_u}{T-(K+1)} $$ And we know that the Cramer Rao bound is $$ \frac{2\sigma^4_u}{T} $$ But as the ML estimator is not unbiased can we still conclude that the OLS estimator is not efficient ? I guess the answer is yes, but then is there an explicit efficient estimator of this variance, as the ML one is not (because it is biased) ?

Am I missing something ?

Answer 1

Under maximum likelihood estimation the estimate of the error variance is given by \begin{equation} \hat \sigma^2 = \frac{\hat{u}^T\hat{u}}{n} \end{equation} As you noted this is not an unbiased estimator, the expected value is \begin{equation} E(\hat \sigma^2) = \sigma^2 \frac{n-k}{n} \end{equation} And the variance is given by \begin{equation} V(\hat \sigma^2) = 2 \sigma^4 \frac{n-k}{n^2} \end{equation} You are correct that the Cramer-Rao (CR) bound is \begin{equation} \frac{2\sigma^4_u}{n} \end{equation} But since the CR bound only applies to unbiased estimators it is possible that the variance of $\hat \sigma^2$ has a smaller variance than the CR bound. Which is true for this case.

For an unibiased estimator of $\sigma^2$ we can use \begin{equation} s^2 = \frac{n}{n-k}\hat \sigma^2 = \frac{\hat{u}^T\hat{u}}{n-k} \end{equation} Which can be shown to be unbiased with variance \begin{equation} V(s^2) = \frac{2 \sigma^4}{n-k} \end{equation} So now we would like to know if this estimator has the smallest variance among unbiased estimators even though it is larger than the CR bound. By using methods developed by Lehman and Sheffe it can be shown that $s^2$ is efficient.

However the mean square error of $\hat \sigma^2$ is smaller than $s^2$, so that while on average $\hat \sigma^2$ is incorrect it tends to be closer to $\sigma^2$.