1
$\begingroup$

I was revisiting my notes about the classical linear regression model, $Y = X \beta$.

If we want to estimate the variance of the least squares estimator we usually suppose that the outputs $y_{i}$ are uncorrelated and have fixed variance $\sigma ^{2}$.

We then have $Var(\hat{\beta} ^{2}) = (X^{T}X)^{-1} \sigma ^{2}$ and we estimate $\sigma^{2}$ using $\hat{\sigma}^{2} = \frac{1}{N-p-1}\sum_{i=1}^{N}{y_{i}-\hat{y_{i}}}$. Where $N$ is the number of training examples and $p$ the number of predictors.

Why does the estimation of the variance of the outputs contains elements from the model ($p$ and $\hat{y}$)?

$\endgroup$
2
  • $\begingroup$ What do you expect it should be? $\endgroup$ Oct 3 at 5:14
  • $\begingroup$ @User1865345 I thought it would be $\frac{1}{N-1} \sum_{i=1}^{N}{(y_{i} - \bar{y})^{2}}$ where $\bar{y}$ is the sample mean. $\endgroup$
    – Toshi Mint
    Oct 3 at 5:30

1 Answer 1

4
$\begingroup$

$\DeclareMathOperator{\X}{\mathbf X^\mathsf T\mathbf X}\DeclareMathOperator{\ep}{\boldsymbol\varepsilon}$

Let $\mathbf M := \mathbf I-\mathbf X(\X)^{-1}\mathbf X^\mathsf T$ be the residual maker. Then

$$\mathbf e =\mathbf M\mathbf y. \tag 1$$

Now $\mathbf e^\mathsf T\mathbf e=\ep^\mathsf T\mathbf M\ep.$ Therefore its expectation

\begin{align}\mathbb E\left[\mathbf e^\mathsf T\mathbf e|\mathbf X\right]&= \mathbb E\left[\ep^\mathsf T\mathbf M\ep|\mathbf X\right]\\&= \mathbb E\left[\operatorname{tr}(\ep^\mathsf T\mathbf M\ep)|\mathbf X\right]\\ &= \mathbb E\left[\operatorname{tr}(\mathbf M\ep\ep^\mathsf T)|\mathbf X\right]\\ &= \operatorname{tr} \left(\mathbf M~\mathbb E\left[(\ep\ep^\mathsf T)|\mathbf X\right]\right)\\&= \operatorname{tr}(\mathbf M~\sigma^2\mathbf I)\\&=(n-P)\sigma^2\tag 2\label 2\end{align}

From $\eqref{2},$ one can deduce a natural (unbiased) estimator of $\sigma^2$ that is $(n-P)^{-1}(\mathbf e^\mathsf T\mathbf e). $


Reference:

$[\rm I]$ Econometric Analysis, William H. Greene, Pearson Education, $2018, $ chapter $4, $ p. $62.$

$\endgroup$
3
  • $\begingroup$ Thanks a lot for your answer. It was very helpful. Unfortunately my vote isn't taken into account because I need at least 15 reputation but it said that my feedback has been recorded. Just a little precision if you permit, may I say that this leads us to deduce an unbiased estimator of $\sigma ^{2}$ taking into account our knowledge of the data, and not the usual unbiased sample variance (because it doesn't incorporate the data)? Because we're conditioning on $\textbf{X}$. $\endgroup$
    – Toshi Mint
    Oct 3 at 15:29
  • $\begingroup$ That's correct assessment. $\endgroup$ Oct 3 at 15:30
  • 1
    $\begingroup$ Thank you again. You have clarified a huge misunderstanding of mine. $\endgroup$
    – Toshi Mint
    Oct 3 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.