# Question about estimating the standard error of the regression- notation and intuition

in a standard linear regression frame work:

$$y_{i}=\beta x_i + \epsilon_i$$

when calculating standard errors, we find an unbiased and consistent estimator of $$var(\hat{\beta})$$. Assume spherical errors. so typically we need :

$$E[\epsilon \epsilon']$$, which is a diagonal matrix. Now if I understand correctly, the element on the $$ith$$ diagonal/entry is then $$E[\epsilon_i \epsilon_i']$$, with the transpose on the second epsilon. we write this as $$\sigma^2$$. but when we go to estimate sigma, we usually use:

$$e'e/(n-k)$$

where e is the residual. Why is it e'e if the diagonal term is $$\epsilon_i \epsilon_i'$$? I think the tranpose (') notation is confusing me.

Assume spherical errors. so typically we need : $$E[\epsilon\epsilon′]$$, which is a diagonal matrix.

It would be nice, but you can never observe $$\epsilon$$. You assume $$E[\epsilon\epsilon′]=\sigma^2I$$.

the element on the ith diagonal/entry is then $$E[\epsilon_i\epsilon′_i]$$, with the transpose on the second epsilon

$$\epsilon_i$$ is a single random variable, so it is equal to $$\epsilon_i'$$.

we write this as σ2 but when we go to estimate sigma, we usually use: $$e′e/(n−k)$$ where e is the residual.

Don't be hasty :)

First step: in a standard linear regression framework, $$y=X\beta+\epsilon$$, $$\epsilon\sim\mathcal{N}(0,\sigma^2I)$$, $$V[y]=V[\epsilon]=\sigma^2I$$.

Second step: $$\hat\beta=(X^TX)^{-1}X^Ty$$, and $$V[\hat\beta]=(X^TX)^{-1}X^TV[y]X(X^TX)^{-1}=(X^TX)^{-1}\sigma^2$$ ($$X^TX$$ is a symmetric matrix.)

Third step: since you can't observe $$\epsilon$$, the best you can do is to use residuals. \begin{align*} e&=y-X\hat\beta=y-X(X^TX)^{-1}X^Ty=y-Hy=(I-H)y \\ E[e]&=E[y]-E[X\hat\beta]=E[y]-X(X^TX)^{-1}X^TE[y]\\ &=X\beta-X(X^TX)^{-1}(X^TX)y=0\\ V[e]&=(I-H)\sigma^2 \end{align*} where $$H=X(X^TX)^{-1}X^T$$ and $$I-H$$ are symmetric and idempotent matrices. The residual sum of squares is: $$RSS=e'e=y^T(I-H)^T(I-H)y=y^T(I-H)y$$ The trace of $$H$$ is equal to the rank of $$X$$, i.e. $$k$$, the number of columns. See https://math.stackexchange.com/questions/1582567/proof-that-trace-of-hat-matrix-in-linear-regression-is-rank-of-x). The trace of $$I-H$$, an $$n\times n$$ matrix, is $$n-k$$.
The residual mean square, $$RMS=\frac{e'e}{n-k}$$ is an unbiased estimator of $$\sigma^2$$: \begin{align*} E[e'e]&\overset{[1]}{=}E[\text{trace}(e'e)]\overset{[2]}{=}E[\text{trace}(ee')]=\text{trace}(E[ee'])\\&=\text{trace}(V[e])=\text{trace}(I-H)\sigma^2=(n-k)\sigma^2\\ E[RMS]&=\frac{E[e'e]}{n-k}=\frac{(n-k)\sigma^2}{n-k}=\sigma^2 \end{align*} So the estimated variance of $$\hat\beta$$ is: $$\hat{V}[\hat\beta]=(X^TX)^{-1}RMS$$ Putting $$S=(X^TX)^{-1}$$, the standard error of $$\hat\beta_j$$ is $$\sqrt{s_{jj}RMS}$$.

[1] $$e'e$$ is a scalar, so $$\text{trace}(e'e)=e'e$$.
[2] If $$e=(a,b,c)$$, then $$e'e=\text{trace}(e'e)=a^2+b^2+c^2$$, and $$ee'=\begin{bmatrix}a \\ b \\ c\end{bmatrix}\begin{bmatrix}a&b&c\end{bmatrix}=\begin{bmatrix}a^2 & ab & ac \\ ab & b^2 & bc \\ ac & ab & c^2\end{bmatrix},\quad\text{trace}(ee')=a^2+b^2+c^2$$