1
$\begingroup$

If we have a linear regression equation $y=X\beta + u$, then we can find the OLS estimate of $\beta$ by minimizing wrt $\hat \beta$: $E(\hat u)=E(y-X\hat\beta )$

However, my textbook suddenly says, out of nowhere, that the OLS estimate of the variance of $u$ (each $u_i$ is iid). $\sigma ^2$ is $\hat \sigma ^2 = \frac {\hat u^T \hat u}{n-K}$, where $n $ is the sample size and $K$ is the amount of independent variables.

I understand that this estimator is unbiased, but I have absolutely no idea how it is derived from the assumption of OLS, or why it is called the OLS estimate of $\sigma$.

How do we derive this estimator?

Improve this question
$\endgroup$
1
  • $\begingroup$ Could this be a duplicate? Have you checked carefully for similar threads? $\endgroup$
    – Richard Hardy
    Nov 1 '17 at 10:50
2
$\begingroup$

The estimator for the variance commonly used in regression does not come from the least squares principle, which only produces an estimate for $\boldsymbol{\beta}$. It is just a bias-corrected version (by the factor $\frac{n}{n-K})$ of the empirical variance

$$\widehat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^n \left(y_i - \mathbf{x}_i^{T} \widehat{\boldsymbol{\beta}} \right)^2$$ which in turn is the maximum likelihood estimator for $\sigma^2$ under the assumption of a normal distribution. It's confusing that many people claim that that is the OLS estimator of the variance.

Improve this answer
$\endgroup$
4
  • $\begingroup$ But isn't it called OLS estimator because it is indirectly based on OLS since its formula contains $\hat \beta$, which is the ols estimate? $\endgroup$
    – user56834
    Nov 1 '17 at 10:25
  • $\begingroup$ The OLS estimator for $\boldsymbol{\beta}$ is identical to the maximum likelihood estimator, under normality. You can then do the same manipulations with the mle and produce an unbiased estimator. That's why, in my opinion, the term "OLS estimator for the variance" is a misnomer. $\endgroup$
    – JohnK
    Nov 1 '17 at 10:28
  • $\begingroup$ But under nonnormality, the maximum likelihood estimator of $\sigma$ is not equal to that formula, yet under nonnormality we can still use the OLS estimator and produce the "OLS estimate" of sigma that way. So perhaps this is a good reason to call it the OLS estimate after all? $\endgroup$
    – user56834
    Nov 1 '17 at 11:47
  • $\begingroup$ Well sure, but my objection is the parameter $\sigma^2$ does not appear directly in the least squares optimization problem. $\endgroup$
    – JohnK
    Nov 1 '17 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.