6
$\begingroup$

The image is a copied and pasted youtube lecture on Linear Regression. I can sort of understand what the lecturer says during the lecture, but I wonder how I actually calculate the $\sigma^2$ in the read box of the image. $\underline{X}$ means a matrix. The linear regression is homoscadastic with no serial correlation.

enter image description here

$\endgroup$
2
  • $\begingroup$ Could you please elaborate on what you mean by "get the $\sigma^2$"? Mathematically, it's there because it was present in the original expression for $\operatorname{Var}(y)$, so there doesn't seem to be anything to explain. $\endgroup$
    – whuber
    Commented Mar 11, 2017 at 19:41
  • 1
    $\begingroup$ You cannot actually calculate $\sigma^2$, because it's a model parameter--you don't know it. You can estimate it in terms of the residuals, but that's not what's going on in the figure you reproduced. $\endgroup$
    – whuber
    Commented Mar 11, 2017 at 19:53

1 Answer 1

10
$\begingroup$

Let lowercase bold letters denote vectors. The linear model is:

$$ y_i = \mathbf{x}_i' \mathbf{b} + \epsilon_i $$

In matrix notation for all observations $i=1, \ldots, n$:

$$\mathbf{y} = X \mathbf{b} + \boldsymbol{\epsilon} $$

The OLS estimator for $\mathbf{b}$ is:

$$\hat{\mathbf{b}} = (X'X)^{-1}X'\mathbf{y} $$

Substituting:

$$\hat{\mathbf{b}} = (X'X)^{-1}X' \left( X \mathbf{b} + \boldsymbol{\epsilon}\right) $$

What's the variance of our estimator $\hat{\mathbf{b}}$? Take the variance (conditional on X)$:

\begin{align*} \operatorname{Var}\left( \hat{\mathbf{b}} \mid X\right) &= \operatorname{Var}\left( (X'X)^{-1}X' \left( X \mathbf{b} + \boldsymbol{\epsilon}\right) \mid X\right) \\ &= \operatorname{Var}\left( (X'X)^{-1}X' \boldsymbol{\epsilon} \mid X\right) \\ &= (X'X)^{-1}X' \operatorname{Var}\left( \boldsymbol{\epsilon} \mid X\right) X(X'X)^{-1} \end{align*}

We assumed that that $\epsilon_i$ were IID. Hence for all $i$ we have $\operatorname{Var}(\epsilon_i) = \sigma^2$, and for the whole vector $\boldsymbol{\epsilon}$ we have $\operatorname{Var}(\boldsymbol{\epsilon}) = \sigma^2 I$ where $I$ is the identity matrix.

Continuing: \begin{align*} \operatorname{Var}\left( \hat{\mathbf{b}} \mid X\right) &= (X'X)^{-1}X' \sigma^2 IX(X'X)^{-1} \\ &= \sigma^2 (X'X)^{-1}X'X(X'X)^{-1} \\ &= \sigma^2 (X'X)^{-1} \end{align*}

$\sigma^2$ is a scalar and can be moved wherever by the commutative property of multiplication. $(X'X)^{-1}X'X = I$ by definition of an inverse matrix.

You can estimate $\sigma^2$ using the residuals from your OLS regression. Let $\hat{e}_i$ be the residual (as opposed to error-term $\epsilon_i$) for observation $i$. That is:

$$ \hat{e}_i = y_i - \mathbf{x}_i' \hat{\mathbf{b}} $$ Note that this is based upon estimate $\hat{\mathbf{b}}$ instead of true values $\mathbf{b}$. Then the usual estimator for $\sigma^2$ is: $$\hat{\sigma}^2 = \frac{1}{n-k} \sum_i \hat{e}_i^2$$ where $k$ is your number of regressors (including a constant).

$\endgroup$
4
  • $\begingroup$ Thank you for your explanation and I think I understand now. $\endgroup$
    – user122358
    Commented Mar 11, 2017 at 20:09
  • $\begingroup$ How would the last part (of estimating sigma) work for ridge regression, assuming k > n? $\endgroup$
    – runr
    Commented Jun 12, 2020 at 3:26
  • 1
    $\begingroup$ Thanks a lot for your answer! Could you please elaborate on how you made this step? Var((𝑋′𝑋)^(−1)𝑋′(𝑋𝐛+𝝐)∣𝑋) = Var((𝑋′𝑋)^(−1)𝑋′𝝐∣𝑋) $\endgroup$ Commented Dec 28, 2021 at 6:35
  • 1
    $\begingroup$ @ Антон Бугаев $X \mathbf{b}$ isn't a random vector: it's a vector of scalars so its variance is 0. It's the same conceptual idea that $\operatorname{Var}(2 + Z) = \operatorname{Var}(Z)$ (where $Z$ is a random variable). $\endgroup$ Commented Dec 29, 2021 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.