1
$\begingroup$

Having consulted a number of sources, I still can't find a complete proof that Regression Sum of Squares ($SS_{regression}$) and ($SS_{residual}$) are independent random variables. I'll be doubly pleased with a proof in matrix form. If it is too involved to be typed up here, I am happy to check it out myself if people can direct me to a book that contains the proof.

$\endgroup$
2
  • 3
    $\begingroup$ stats.stackexchange.com/questions/117406/… should be helpful $\endgroup$ – Christoph Hanck Feb 19 at 6:49
  • $\begingroup$ I see. Is the idea that $SS_{regression}$ is a function of $\hat{y}$ and $SS_{residual}$ a function of $\hat{\epsilon}$ and thus conclude $SS_{residual}$ is independent of $SS_{regression}$ once we can show that $\hat{y}$ and $\hat{\epsilon}$ are uncorrelated, given the assumption of marginally normal errors, with constant variance? $\endgroup$ – ColorStatistics Feb 19 at 13:45
2
$\begingroup$

Assume $y \sim \operatorname{Normal}(\beta X, \Sigma)$ with constant diagonal covariance $\operatorname{Cov}(y)=\Sigma=\sigma^2 \mathbb I$ and mean $\bar y=\mu$. Using the hat matrix $\mathbb H$ we have that:

$$\hat y =\mathbb H y$$

And

$$\epsilon=y-\hat y=(\mathbb I -\mathbb H)y$$

Then (because the hat matrix is idempotent $\mathbb H^2 = \mathbb H$) $$\begin{cases} SSR = (\hat y - \mu)^T(\hat y - \mu)= y^T \mathbb Hy \color{red}{+\mu^2-2\mu\mathbf 1^T\hat y}= y^T \mathbb Hy \color{red}{-(2n-1)\mu^2}\\ SSE = \epsilon ^T \epsilon = y^T (\mathbb I -\mathbb H)y \end{cases}$$

$$\operatorname{Cov}(SSR,SSE) = \operatorname{Cov}(y^T \mathbb Hy,y^T (\mathbb I -\mathbb H)y)$$

Using that $\operatorname{Cov}(x^TAx,x^TBx)=4\mu^TA\Sigma B\mu + 2\operatorname{tr}(A\Sigma B\Sigma)$ (see Prove that $\mathrm{Cov}(x^TAx,x^TBx) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \mu^TA \Sigma B \mu$)

$$\operatorname{Cov}(SSR,SSE) =4\mu^T\mathbb H\Sigma (\mathbb I -\mathbb H)\mu + 2\operatorname{tr}(\mathbb H\Sigma (\mathbb I -\mathbb H)\Sigma)\\ $$

Since $\Sigma=\sigma^2 \mathbb I$ is constant diagonal:

$$\operatorname{Cov}(SSR,SSE) =4\sigma^2\mu^T\mathbb H (\mathbb I -\mathbb H)\mu + 2\sigma^4\operatorname{tr}(\mathbb H (\mathbb I -\mathbb H))\\ =4\sigma^2\mu^T (\mathbb H -\mathbb H^2)\mu +2\sigma^4\operatorname{tr}(\mathbb H -\mathbb H^2)\\ =0$$

$\endgroup$
11
  • $\begingroup$ Very elegant proof. I like it but I am not clear on something and not convinced for the case when x is non-random. As I see it, key to this proof is the assumption that $y$~$N(0,\sum)$. Given that the dependent variable in our sample is what it is and its population average is unknown, what does it mean to assume that the random vector y is multivariate normally distributed with mean 0 and constant variance. Don't we make assumptions only about the distribution of the error term and derive the distribution of the random vector y? $\endgroup$ – ColorStatistics Feb 19 at 16:21
  • $\begingroup$ When x is non-random, the random vector y will, by design, not have mean 0. In fact the $y_i$ will not all have the same mean. Would you disagree? $\endgroup$ – ColorStatistics Feb 19 at 16:21
  • $\begingroup$ @ColorStatistics the zero mean part is not that important, it's just to simplify the expressions, but the result would be the same (a mean term would appear in the Cov expression, but then it would be nullified by H and H²) $\endgroup$ – Firebug Feb 19 at 16:24
  • $\begingroup$ @ColorStatistics what do you mean by non-random y? With covariance zero then it's even simpler $\endgroup$ – Firebug Feb 19 at 16:24
  • $\begingroup$ I see. Thank you for the clarification. I am clear on the case when X is random. But for x non-random, the $y_i$ will have different means so we need a different approach. See this answer on the fact that $y_i$ will have different means when x non-random. stats.stackexchange.com/a/308493/198058 $\endgroup$ – ColorStatistics Feb 19 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.