0
$\begingroup$

in a standard linear regression frame work:

$y_{i}=\beta x_i + \epsilon_i$

when calculating standard errors, we find an unbiased and consistent estimator of $var(\hat{\beta})$. Assume spherical errors. so typically we need :

$E[\epsilon \epsilon']$, which is a diagonal matrix. Now if I understand correctly, the element on the $ith$ diagonal/entry is then $E[\epsilon_i \epsilon_i']$, with the transpose on the second epsilon. we write this as $\sigma^2$. but when we go to estimate sigma, we usually use:

$e'e/(n-k)$

where e is the residual. Why is it e'e if the diagonal term is $\epsilon_i \epsilon_i'$? I think the tranpose (') notation is confusing me.

$\endgroup$
1
$\begingroup$

Assume spherical errors. so typically we need : $E[\epsilon\epsilon′]$, which is a diagonal matrix.

It would be nice, but you can never observe $\epsilon$. You assume $E[\epsilon\epsilon′]=\sigma^2I$.

the element on the ith diagonal/entry is then $E[\epsilon_i\epsilon′_i]$, with the transpose on the second epsilon

$\epsilon_i$ is a single random variable, so it is equal to $\epsilon_i'$.

we write this as σ2 but when we go to estimate sigma, we usually use: $e′e/(n−k)$ where e is the residual.

Don't be hasty :)

First step: in a standard linear regression framework, $y=X\beta+\epsilon$, $\epsilon\sim\mathcal{N}(0,\sigma^2I)$, $V[y]=V[\epsilon]=\sigma^2I$.

Second step: $\hat\beta=(X^TX)^{-1}X^Ty$, and $$V[\hat\beta]=(X^TX)^{-1}X^TV[y]X(X^TX)^{-1}=(X^TX)^{-1}\sigma^2$$ ($X^TX$ is a symmetric matrix.)

Third step: since you can't observe $\epsilon$, the best you can do is to use residuals. \begin{align*} e&=y-X\hat\beta=y-X(X^TX)^{-1}X^Ty=y-Hy=(I-H)y \\ E[e]&=E[y]-E[X\hat\beta]=E[y]-X(X^TX)^{-1}X^TE[y]\\ &=X\beta-X(X^TX)^{-1}(X^TX)y=0\\ V[e]&=(I-H)\sigma^2 \end{align*} where $H=X(X^TX)^{-1}X^T$ and $I-H$ are symmetric and idempotent matrices. The residual sum of squares is: $$RSS=e'e=y^T(I-H)^T(I-H)y=y^T(I-H)y$$ The trace of $H$ is equal to the rank of $X$, i.e. $k$, the number of columns. See https://math.stackexchange.com/questions/1582567/proof-that-trace-of-hat-matrix-in-linear-regression-is-rank-of-x). The trace of $I-H$, an $n\times n$ matrix, is $n-k$.
The residual mean square, $$RMS=\frac{e'e}{n-k}$$ is an unbiased estimator of $\sigma^2$: \begin{align*} E[e'e]&\overset{[1]}{=}E[\text{trace}(e'e)]\overset{[2]}{=}E[\text{trace}(ee')]=\text{trace}(E[ee'])\\&=\text{trace}(V[e])=\text{trace}(I-H)\sigma^2=(n-k)\sigma^2\\ E[RMS]&=\frac{E[e'e]}{n-k}=\frac{(n-k)\sigma^2}{n-k}=\sigma^2 \end{align*} So the estimated variance of $\hat\beta$ is: $$\hat{V}[\hat\beta]=(X^TX)^{-1}RMS$$ Putting $S=(X^TX)^{-1}$, the standard error of $\hat\beta_j$ is $\sqrt{s_{jj}RMS}$.


[1] $e'e$ is a scalar, so $\text{trace}(e'e)=e'e$.
[2] If $e=(a,b,c)$, then $e'e=\text{trace}(e'e)=a^2+b^2+c^2$, and $$ee'=\begin{bmatrix}a \\ b \\ c\end{bmatrix}\begin{bmatrix}a&b&c\end{bmatrix}=\begin{bmatrix}a^2 & ab & ac \\ ab & b^2 & bc \\ ac & ab & c^2\end{bmatrix},\quad\text{trace}(ee')=a^2+b^2+c^2$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.