# Proving that $(\hat{\beta} - \beta)' (X' X) (\hat{\beta} - \beta)$ is independent with SSE

Exercise:

Prove that $$\mathbf{(\hat{\beta} - \beta)' (X' X) (\hat{\beta} - \beta)}$$ and SSE are independent for a Least Squares Regression Model.

Attempt:

Note that by $$'$$ I denote the transpose matrix of the given quantity. I will also use the notation $$\beta: = b$$ for the sake of simplification to the code.

I tried, first of all, to yield a matrix expression for SSE so that it can be useful to make conclusions and calculations with the matrix expressions given :

\begin{align*} \text{SSE} &= \mathbf{e'e = (y-X\hat{b})'(y-X\hat{b})}\\ &= \mathbf{y'y - 2\hat{b}'X'y + \hat{b}'X'X\hat{b}}\\ &= \mathbf{y'y - 2\hat{b}'X'y + \hat{b}'X'X(X'X)^{-1}X'y}\\ &= \mathbf{y'y - 2\hat{b}'X'y+\hat{b}'IX'y}\\ &= \mathbf{y'y - \hat{b}'X'y}\\ \end{align*}

I assume that now, to show the independance, I need to show that the Covariance of them is equal to zero, namingly :

$$\text{Cov}\Big(\mathbf{(\hat{b} - b)' (X' X) (\hat{b} - b), \mathbf{y'y - \hat{b}'X'y}}\Big)$$ $$=$$ $$\mathbb{E}\Big(\mathbf{(\hat{b} - b)' (X' X) (\hat{b} - b)\big(\mathbf{y'y - \hat{b}'X'y}\big)\Big)}$$

$$-\mathbb{E}\Big(\mathbf{(\hat{b} - b)' (X' X) (\hat{b} - b)}\Big)\mathbb{E}\Big(\mathbf{y'y - \hat{b}'X'y}\Big)$$

But this seems very, very complicated and I cannot see how to continue. I don't even know if this approach is correct or if there exists a simpler way to show the independance between the two quantities given.

Any hints or even better thorough elaborations will be very much appreciated.

• Hint: one is a function of $HY$, the other is a function of $(I-H)Y$, where $H$ is the hat matrix $X(X^TX)^{-1}X^T$. Under iid Gaussian errors, these two pieces are independent, as it is easy to check they are jointly Gaussian with zero covariance. This implies the desired independence. – guy Nov 29 '18 at 15:09
• @guy Hi, interesting approach (seems clever as well). How does one rigorously elaborate the given hint though ? – Rebellos Nov 29 '18 at 16:35
• @Rebellos what part is unclear? To check independence it suffices (for jointly Gaussian data) to check the covariance matrix is 0. This relies on the fact that $H$ is symmetric and idempotent. Writing the two expressions as functions of $HY$ and $(I - H)Y$ is just definitions and algebra (but note that $X\widehat \beta = HY$). – guy Nov 29 '18 at 19:58
• @guy I've grasped all that, the only thing I cannot see implicitly (because I am not that experienced with these expressions) is how I would write them as functions of the quantities mentioned. The rest would be trivial. – Rebellos Nov 29 '18 at 20:00
• @Rebellos $Y - X\widehat \beta$ is $(I - H) Y$, while $X(\widehat \beta - \beta)$ is $(HY - X\beta)$. – guy Nov 29 '18 at 22:34

Since $$[X(\hat\beta-\beta)]^TX(\hat\beta-\beta)$$ is a function of $$X(\hat\beta-\beta)$$ and $$\operatorname{SSE}:=(Y-X\hat\beta)^T(Y-X\hat\beta)$$ is a function of $$Y-X\hat\beta$$, to prove independence it's enough to show that the random vectors $$Y-X\hat\beta$$ and $$X(\hat\beta-\beta)$$ are jointly Gaussian and have zero covariance.
Introduce the hat matrix $$H:=X(X^TX)^{-1}X^T$$. Check that $$H^2=H$$ and $$HX=X$$ and $$HY=X\hat\beta$$, where $$\hat\beta:=(X^TX)^{-1}X^TY$$ is the vector of estimated parameters. These facts imply that $$X\hat\beta=HY=H(X\beta+\epsilon)=X\beta+H\epsilon\tag1$$ and $$Y-X\hat\beta=Y-HY=(I-H)Y=(I-H)(X\beta +\epsilon)=(I-H)\epsilon.\tag2$$ From (2) and (1) it's clear that $$Y-X\hat\beta$$ and $$X(\hat\beta-\beta)$$ are jointly Gaussian. Moreover, we can easily compute their covariance: \begin{align} \operatorname{Cov}\big(Y-X\hat\beta,X(\hat\beta-\beta)\big) &\stackrel{(2,1)}=\operatorname{Cov}\big((I-H)\epsilon,H\epsilon\big)\\ &\stackrel{(3)}=E[(I-H)\epsilon(H\epsilon)^T]\\ &\stackrel{(4)}=(I-H)E(\epsilon\epsilon^T)H^T\\ &\stackrel{(5)}=\sigma^2 (I-H)H\\ &\stackrel{(6)}=0. \end{align} In (3) we use the fact that $$E(\epsilon)=0$$. In (4) we see that everything except $$\epsilon\epsilon^ T$$ is non-stochastic. In (5) we use $$E(\epsilon\epsilon^ T)=\sigma^2I$$ and that $$H$$ is symmetric. In (6) we use the idempotency property $$H^2=H$$.
Note: The independence of the random vectors $$Y-X\hat\beta$$ and $$X(\hat\beta-\beta)$$ is distinct from their (pointwise) orthogonality, which is proved as follows: $$\big(Y-X\hat\beta\big)^T\big(X(\hat\beta-\beta)\big)= \big((I-H)\epsilon\big)^ T\big(H\epsilon\big)=\epsilon^T\underbrace{(I-H)H}_0\epsilon$$