0
$\begingroup$

Suppose I have a simple linear regression model: $$ y_i = \alpha + \beta x_i + \epsilon_i, $$ and I know the expression for the least square estimators of $\alpha$ and $\beta$: $$ \hat{\alpha} = \bar{y} - \hat{\beta}\bar{x},\\ \hat{\beta} = S^{-1}_{xx}S_{xy} = \left(\sum^n_{i=1}(x_i-\bar{x})^2 \right)^{-1}\sum^n_{i=1}(x_i-\bar{x})(y_i-\bar{y}). $$ And the sum of square of residual is given by: $$ RSS = \sum^n_{i=1}\hat{\epsilon}_i^2 = \sum^n_{i=1}(y_i-\hat{y}_i)^2. $$ I was told that $\hat{\alpha}$ and $\hat{\beta}$ are independent of $RSS$, but I can't figure out why.

$\endgroup$
1
  • $\begingroup$ Following this answer, we have $Z_1=\sqrt n\,\overline y\,,Z_2=\hat\beta\sqrt{s_{xx}}$ and $RSS=\sum_{i=3}^n Z_i^2$ where the $Z_i$'s are all independently distributed normal random variables. This shows the independence of $\hat\beta$ and $RSS$. Now $\hat\alpha=\overline y-\hat\beta\overline x$ is a function of $Z_1,Z_2$ only which is therefore also independent of $RSS$. A general proof is shown here: stats.stackexchange.com/q/173396/119261. $\endgroup$ Mar 27, 2021 at 19:20

1 Answer 1

0
$\begingroup$

It is easiest if you take a geometric approach. The coefficient vector and the residual are independent if they are orthogonal.

$$ (y - \hat{y})^T \beta $$

If you replace $\beta$ by the projection matrix and work through the algebra, what do you get?

$\endgroup$
1
  • $\begingroup$ I understand that the coefficient vector and the residual are independent. I am just wondering how do I prove the independence for the sum of square of residuals. Besides, what do you mean by replace $\beta$ by the projection matrix? $\endgroup$
    – Van Tom
    Oct 8, 2020 at 4:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.