5
$\begingroup$

Having consulted a number of sources, I still can't find a complete proof that Regression Sum of Squares ($SS_{regression}$) and ($SS_{residual}$) are independent random variables. I'll be doubly pleased with a proof in matrix form. If it is too involved to be typed up here, I am happy to check it out myself if people can direct me to a book that contains the proof.

$\endgroup$
2
  • 3
    $\begingroup$ stats.stackexchange.com/questions/117406/… should be helpful $\endgroup$ Feb 19, 2021 at 6:49
  • $\begingroup$ I see. Is the idea that $SS_{regression}$ is a function of $\hat{y}$ and $SS_{residual}$ a function of $\hat{\epsilon}$ and thus conclude $SS_{residual}$ is independent of $SS_{regression}$ once we can show that $\hat{y}$ and $\hat{\epsilon}$ are uncorrelated, given the assumption of marginally normal errors, with constant variance? $\endgroup$ Feb 19, 2021 at 13:45

1 Answer 1

3
$\begingroup$

Assume $y \sim \operatorname{Normal}(\beta X, \Sigma)$ with constant diagonal covariance $\operatorname{Cov}(y)=\Sigma=\sigma^2 \mathbb I$ and mean $\bar y=\mu$. Using the hat matrix $\mathbb H$ we have that:

$$\hat y =\mathbb H y$$

And

$$\epsilon=y-\hat y=(\mathbb I -\mathbb H)y$$

Then (because the hat matrix is idempotent $\mathbb H^2 = \mathbb H$) $$\begin{cases} SSR = (\hat y - \mu)^T(\hat y - \mu)= y^T \mathbb Hy \color{red}{+n\mu^2-2\mu\mathbf 1^T\hat y}= y^T \mathbb Hy \color{red}{-n\mu^2}\\ SSE = \epsilon ^T \epsilon = y^T (\mathbb I -\mathbb H)y \end{cases}$$

$$\operatorname{Cov}(SSR,SSE) = \operatorname{Cov}(y^T \mathbb Hy,y^T (\mathbb I -\mathbb H)y)$$

Using that $\operatorname{Cov}(x^TAx,x^TBx)=4\mu^TA\Sigma B\mu + 2\operatorname{tr}(A\Sigma B\Sigma)$ (see Prove that $\mathrm{Cov}(x^TAx,x^TBx) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \mu^TA \Sigma B \mu$)

$$\operatorname{Cov}(SSR,SSE) =4\mu^T\mathbb H\Sigma (\mathbb I -\mathbb H)\mu + 2\operatorname{tr}(\mathbb H\Sigma (\mathbb I -\mathbb H)\Sigma)\\ $$

Since $\Sigma=\sigma^2 \mathbb I$ is constant diagonal:

$$\operatorname{Cov}(SSR,SSE) =4\sigma^2\mu^T\mathbb H (\mathbb I -\mathbb H)\mu + 2\sigma^4\operatorname{tr}(\mathbb H (\mathbb I -\mathbb H))\\ =4\sigma^2\mu^T (\mathbb H -\mathbb H^2)\mu +2\sigma^4\operatorname{tr}(\mathbb H -\mathbb H^2)\\ =0$$

$\endgroup$
16
  • $\begingroup$ Very elegant proof. I like it but I am not clear on something and not convinced for the case when x is non-random. As I see it, key to this proof is the assumption that $y$~$N(0,\sum)$. Given that the dependent variable in our sample is what it is and its population average is unknown, what does it mean to assume that the random vector y is multivariate normally distributed with mean 0 and constant variance. Don't we make assumptions only about the distribution of the error term and derive the distribution of the random vector y? $\endgroup$ Feb 19, 2021 at 16:21
  • $\begingroup$ When x is non-random, the random vector y will, by design, not have mean 0. In fact the $y_i$ will not all have the same mean. Would you disagree? $\endgroup$ Feb 19, 2021 at 16:21
  • 3
    $\begingroup$ Hi, nice post. Just a quick comment. That $Hy$ and $(I-H)y$ are uncorrelated follows immediately since $H$ and $I-H$ are orthogonal. Due to joint normality, we know $Hy$ and $(I-H)y$ are independent. After that, we know that any functions of those are independent, including their sums of squares, giving the result. $\endgroup$
    – user257566
    Apr 9, 2021 at 21:25
  • 1
    $\begingroup$ I see, thanks for explaining. FWIW, your post didn't prove that the sums of squares were independent, just that they were uncorrelated. Am I misreading? $\endgroup$
    – user257566
    Apr 9, 2021 at 21:34
  • 1
    $\begingroup$ @user257566 No, you got it right. Now that you mentioned it, I had it in my mind that uncorrelated Chi-square variables are independent, though I'm not so sure now. I'll have to check though their relationship to squared Normals probably helps. Perhaps it brings back to your initial assessment. $\endgroup$
    – Firebug
    Apr 10, 2021 at 1:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.