0
$\begingroup$

Suppose I have a simple linear regression model: $$ y_i = \alpha + \beta x_i + \epsilon_i, $$ and I know the expression for the least square estimators of $\alpha$ and $\beta$: $$ \hat{\alpha} = \bar{y} - \hat{\beta}\bar{x},\\ \hat{\beta} = S^{-1}_{xx}S_{xy} = \left(\sum^n_{i=1}(x_i-\bar{x})^2 \right)^{-1}\sum^n_{i=1}(x_i-\bar{x})(y_i-\bar{y}). $$ And the sum of square of residual is given by: $$ RSS = \sum^n_{i=1}\hat{\epsilon}_i^2 = \sum^n_{i=1}(y_i-\hat{y}_i)^2. $$ I was told that $\hat{\alpha}$ and $\hat{\beta}$ are independent of $RSS$, but I can't figure out why.

$\endgroup$
1
  • $\begingroup$ Following this answer, we have $Z_1=\sqrt n\,\overline y\,,Z_2=\hat\beta\sqrt{s_{xx}}$ and $RSS=\sum_{i=3}^n Z_i^2$ where the $Z_i$'s are all independently distributed normal random variables. This shows the independence of $\hat\beta$ and $RSS$. Now $\hat\alpha=\overline y-\hat\beta\overline x$ is a function of $Z_1,Z_2$ only which is therefore also independent of $RSS$. A general proof is shown here: stats.stackexchange.com/q/173396/119261. $\endgroup$ – StubbornAtom Mar 27 at 19:20
0
$\begingroup$

It is easiest if you take a geometric approach. The coefficient vector and the residual are independent if they are orthogonal.

$$ (y - \hat{y})^T \beta $$

If you replace $\beta$ by the projection matrix and work through the algebra, what do you get?

$\endgroup$
1
  • $\begingroup$ I understand that the coefficient vector and the residual are independent. I am just wondering how do I prove the independence for the sum of square of residuals. Besides, what do you mean by replace $\beta$ by the projection matrix? $\endgroup$ – Van Tom Oct 8 '20 at 4:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.