4
$\begingroup$

In Elements of Statistical Learning, pg 47, at the very bottom, it states that $\hat{\beta}$ and $\hat{\sigma}^2$ are statistically independent.

Is this saying that they are independent when conditioned on $X$, or is it saying they're marginally independent? If it is saying they're marginally independent, which seems to be the case in the book, I'm confused because it would seem they're not marginally independent.

We know that $$ \hat{\beta} = (X^TX)^{-1}X^Ty$$

$$\hat{\sigma}^2 = \frac{1}{N - p - 1} ||y - \hat{y} ||_2^2 = \frac{1}{N - p - 1} ||y - X\hat{\beta} ||_2^2 $$

By inspection, $\hat{\sigma}^2$ is a function of $\hat{\beta}$, so it's not obvious how the two are marginally independent.

$\endgroup$
5
  • 1
    $\begingroup$ Hi: Here's a proof but it's not trivial. stats.stackexchange.com/questions/286179/… $\endgroup$
    – mlofton
    May 10 at 14:16
  • 1
  • $\begingroup$ For additional demonstrations, see posts about Cochran's Theorem. $\endgroup$
    – whuber
    May 10 at 14:59
  • $\begingroup$ @StubbornAtom I looked at the first link, and I am confused how the accepted answer shows this. It shows that the residuals and $\hat{\beta}$ are uncorrelated, but how do we know that $\hat{\beta}$ and the residuals are jointly normal? I think that step is missing. I know that once we show that two variables are jointly normal, then uncorrelated implies independence. $\endgroup$ May 10 at 15:39
  • $\begingroup$ @StubbornAtom I think I understand now. We need to show that any linear combination of the residuals and $\hat{\beta}$ is normal. The residual and $\hat{\beta}$ are $\epsilon$ scaled plus some constant. So any linear combination of the two is also $\epsilon$ scaled plus some constant. And since we assume $\epsilon$ is normally distributed, any scaling plus shift of it is also normally distributed, so this implies the residuals and $\hat{\beta}$ are jointly normal, hence showing they're uncorrelated means they're independent. $\endgroup$ May 10 at 15:49