8
$\begingroup$

An important basic theorem of linear regression is that the maximum likelihood estimates (MLEs) of the coefficients coincide with their least-squares estimates.

What about the variance of the error term? Does its MLE coincide with the variance of the residuals, that is,

$$ \frac{∑_i (Y_i - \hat Y_i)^2}{n} $$

where the $Y_i$s are the observed values of the dependent variable, the $\hat Y_i$s are the fitted values, and $n$ is the sample size?

$\endgroup$
7
  • $\begingroup$ Question is unclear -- what's this "variance of the residuals" you mention? Please give an unambiguous estimator of $\sigma^2$ $\endgroup$
    – Glen_b
    Commented Sep 15, 2017 at 2:13
  • $\begingroup$ @Glen_b Edited. I thought that was the only thing "variance of the residuals" could mean. $\endgroup$ Commented Sep 15, 2017 at 14:22
  • $\begingroup$ which sample variance ... Bessel-corrected? not-Bessel-corrected? Or corrected for the df of the error term in the regression? $\endgroup$
    – Glen_b
    Commented Sep 16, 2017 at 8:30
  • $\begingroup$ @Glen_b Edited again. $\endgroup$ Commented Sep 16, 2017 at 14:22
  • 1
    $\begingroup$ Many, many people would insist that "the" sample variance is the Bessel-corrected one. (I don't agree with them, I think that the $n$-denominator is a possibility.) ... and at least some people that call the "n-1" version "the" sample variance ordinarily would insist that with regression "the" sample variance would be the df-corrected one there also. All said and done, it's best to be aware of the varying expectations. What do you see when you type "the sample variance" (including quotes) into say Google? For me it shows a big picture of the "n-1" form. $\endgroup$
    – Glen_b
    Commented Sep 16, 2017 at 22:07

1 Answer 1

10
$\begingroup$

If $Y \sim \mathcal N(X\beta, \sigma^2 I)$ then the log likelihood is $$ l(\beta, \sigma^2|y) = -\frac n2 \log (2\pi) - \frac n2 \log(\sigma^2) - \frac 1{2\sigma^2}||y-X\beta||^2 $$ and assuming non-stochastic and full rank predictors. From this we find that $$ \frac{\partial l}{\partial \sigma^2} = 0 \implies \hat \sigma^2 = \frac 1n ||Y-X\hat \beta||^2. $$

We want to know when the MLE $\hat \sigma^2$ is equal to the sample variance of the residuals $$ \tilde \sigma^2 = \frac{1}{n}\sum_{i=1}^n (e_i - \bar e)^2 $$

where $e = Y - \hat Y$ are the residuals. We know $$ n\tilde \sigma^2 = e^Te - n\bar e^2 $$ while $$ n\hat \sigma^2 = e^Te $$ so this tells us the two are equal when the constant vector $\mathbf 1$ is in the column space of $X$, which means $\bar e = 0$. If that is not the case then the two won't be exactly equal.


I'm leaving the rest of my answer here but as I understand OP's question better I don't think it applies.

Note $$ ||Y - X\hat \beta||^2 = (Y - HY)^T(Y - HY) = Y^T(I-H)Y $$ where $H = X(X^TX)^{-1}X^T$. This means that we have a Gaussian quadratic form, so $$ Var\left(Y^T (I-H)Y\right) = 2\sigma^4 \text{tr}(I-H) + 4\sigma^2 \beta^T X^T(I-H)X\beta. $$ $X^T(I-H)X = X^TX - X^TX(X^TX)^{-1}X^TX = 0$ and $\text{tr}(I-H) = n-p$ so we have $$ Var(\hat \sigma^2) = \frac{2\sigma^4(n-p)}{n^2}. $$

The standard estimate of $\sigma^2$ is probably $\tilde \sigma^2 := \frac{1}{n-p}||Y - X\hat \beta||^2$ (which is unbiased, as we can see by computing $E\left(Y^T(I-H)Y\right)$) so $$ Var(\tilde \sigma^2) = \frac{2\sigma^4}{n-p}. $$

I'm not entirely sure what more than this you're looking for, as technically what you asked for was the variance of the residuals which is $$ Var(e) = Var\left((I-H)Y\right) =\sigma^2 (I-H) $$ but I don't think that's what you mean. Or if that is what you mean, then we can directly compare this to $Var(\varepsilon) = \sigma^2 I$ and the difference comes down to $\sigma^2 H$.

$\endgroup$
5
  • 1
    $\begingroup$ (+1) From my interpretation of the question, OP was looking for the first 4 lines of your answer. In this case, the MLE of the variance is indeed the (sample) variance of the residuals (times $\frac{n}{n-1}$) $\endgroup$
    – user795305
    Commented Sep 15, 2017 at 3:40
  • $\begingroup$ (+1) The line beginning with $\tfrac{\partial l}{\partial \sigma^2}$ seems to answer my question in the affirmative. I'm not sure why the residuals would be $(I - H)Y$. You might find my edit to the question helpful. $\endgroup$ Commented Sep 15, 2017 at 17:14
  • $\begingroup$ @Kodiologist i've added an update that i think actually answers your question. Please let me know if there's still something unaddressed $\endgroup$
    – jld
    Commented Sep 27, 2017 at 21:00
  • $\begingroup$ Looks good, thanks. $\bar e = 0$ ought to hold by the construction of $\hat Y$ so long as all the usual model assumptions are correct. $\endgroup$ Commented Sep 27, 2017 at 22:09
  • 1
    $\begingroup$ @Kodiologist We always say $\bar e = 0$ but that's under the implicit assumption of an intercept and in total generality we can't count on that $\endgroup$
    – jld
    Commented Sep 27, 2017 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.