Prove that $ \mathbf{(\hat{\beta} - \beta)' (X' X) (\hat{\beta} - \beta)}$ and SSE are independent for a Least Squares Regression Model.


Note that by $'$ I denote the transpose matrix of the given quantity. I will also use the notation $ \beta: = b$ for the sake of simplification to the code.

I tried, first of all, to yield a matrix expression for SSE so that it can be useful to make conclusions and calculations with the matrix expressions given :

\begin{align*} \text{SSE} &= \mathbf{e'e = (y-X\hat{b})'(y-X\hat{b})}\\ &= \mathbf{y'y - 2\hat{b}'X'y + \hat{b}'X'X\hat{b}}\\ &= \mathbf{y'y - 2\hat{b}'X'y + \hat{b}'X'X(X'X)^{-1}X'y}\\ &= \mathbf{y'y - 2\hat{b}'X'y+\hat{b}'IX'y}\\ &= \mathbf{y'y - \hat{b}'X'y}\\ \end{align*}

I assume that now, to show the independance, I need to show that the Covariance of them is equal to zero, namingly :

$$\text{Cov}\Big(\mathbf{(\hat{b} - b)' (X' X) (\hat{b} - b), \mathbf{y'y - \hat{b}'X'y}}\Big)$$ $$=$$ $$\mathbb{E}\Big(\mathbf{(\hat{b} - b)' (X' X) (\hat{b} - b)\big(\mathbf{y'y - \hat{b}'X'y}\big)\Big)}$$

$$-\mathbb{E}\Big(\mathbf{(\hat{b} - b)' (X' X) (\hat{b} - b)}\Big)\mathbb{E}\Big(\mathbf{y'y - \hat{b}'X'y}\Big)$$

But this seems very, very complicated and I cannot see how to continue. I don't even know if this approach is correct or if there exists a simpler way to show the independance between the two quantities given.

Any hints or even better thorough elaborations will be very much appreciated.

  • 1
    $\begingroup$ Hint: one is a function of $HY$, the other is a function of $(I-H)Y$, where $H$ is the hat matrix $X(X^TX)^{-1}X^T$. Under iid Gaussian errors, these two pieces are independent, as it is easy to check they are jointly Gaussian with zero covariance. This implies the desired independence. $\endgroup$ – guy Nov 29 '18 at 15:09
  • 1
    $\begingroup$ @guy Hi, interesting approach (seems clever as well). How does one rigorously elaborate the given hint though ? $\endgroup$ – Rebellos Nov 29 '18 at 16:35
  • $\begingroup$ @Rebellos what part is unclear? To check independence it suffices (for jointly Gaussian data) to check the covariance matrix is 0. This relies on the fact that $H$ is symmetric and idempotent. Writing the two expressions as functions of $HY$ and $(I - H)Y$ is just definitions and algebra (but note that $X\widehat \beta = HY$). $\endgroup$ – guy Nov 29 '18 at 19:58
  • $\begingroup$ @guy I've grasped all that, the only thing I cannot see implicitly (because I am not that experienced with these expressions) is how I would write them as functions of the quantities mentioned. The rest would be trivial. $\endgroup$ – Rebellos Nov 29 '18 at 20:00
  • $\begingroup$ @Rebellos $Y - X\widehat \beta$ is $(I - H) Y$, while $X(\widehat \beta - \beta)$ is $(HY - X\beta)$. $\endgroup$ – guy Nov 29 '18 at 22:34

Since $[X(\hat\beta-\beta)]^TX(\hat\beta-\beta)$ is a function of $X(\hat\beta-\beta)$ and $\operatorname{SSE}:=(Y-X\hat\beta)^T(Y-X\hat\beta)$ is a function of $Y-X\hat\beta$, to prove independence it's enough to show that the random vectors $Y-X\hat\beta$ and $X(\hat\beta-\beta)$ are jointly Gaussian and have zero covariance.

Introduce the hat matrix $ H:=X(X^TX)^{-1}X^T$. Check that $H^2=H$ and $HX=X$ and $HY=X\hat\beta$, where $\hat\beta:=(X^TX)^{-1}X^TY$ is the vector of estimated parameters. These facts imply that $$X\hat\beta=HY=H(X\beta+\epsilon)=X\beta+H\epsilon\tag1 $$ and $$Y-X\hat\beta=Y-HY=(I-H)Y=(I-H)(X\beta +\epsilon)=(I-H)\epsilon.\tag2 $$ From (2) and (1) it's clear that $Y-X\hat\beta$ and $X(\hat\beta-\beta)$ are jointly Gaussian. Moreover, we can easily compute their covariance: $$\begin{align} \operatorname{Cov}\big(Y-X\hat\beta,X(\hat\beta-\beta)\big) &\stackrel{(2,1)}=\operatorname{Cov}\big((I-H)\epsilon,H\epsilon\big)\\ &\stackrel{(3)}=E[(I-H)\epsilon(H\epsilon)^T]\\ &\stackrel{(4)}=(I-H)E(\epsilon\epsilon^T)H^T\\ &\stackrel{(5)}=\sigma^2 (I-H)H\\ &\stackrel{(6)}=0. \end{align} $$ In (3) we use the fact that $E(\epsilon)=0$. In (4) we see that everything except $\epsilon\epsilon^ T$ is non-stochastic. In (5) we use $E(\epsilon\epsilon^ T)=\sigma^2I$ and that $H$ is symmetric. In (6) we use the idempotency property $H^2=H$.

Note: The independence of the random vectors $Y-X\hat\beta$ and $X(\hat\beta-\beta)$ is distinct from their (pointwise) orthogonality, which is proved as follows: $$\big(Y-X\hat\beta\big)^T\big(X(\hat\beta-\beta)\big)= \big((I-H)\epsilon\big)^ T\big(H\epsilon\big)=\epsilon^T\underbrace{(I-H)H}_0\epsilon $$

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.