Suppose I have a simple linear regression model: $$ y_i = \alpha + \beta x_i + \epsilon_i, $$ and I know the expression for the least square estimators of $\alpha$ and $\beta$: $$ \hat{\alpha} = \bar{y} - \hat{\beta}\bar{x},\\ \hat{\beta} = S^{-1}_{xx}S_{xy} = \left(\sum^n_{i=1}(x_i-\bar{x})^2 \right)^{-1}\sum^n_{i=1}(x_i-\bar{x})(y_i-\bar{y}). $$ And the sum of square of residual is given by: $$ RSS = \sum^n_{i=1}\hat{\epsilon}_i^2 = \sum^n_{i=1}(y_i-\hat{y}_i)^2. $$ I was told that $\hat{\alpha}$ and $\hat{\beta}$ are independent of $RSS$, but I can't figure out why.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Following this answer, we have $Z_1=\sqrt n\,\overline y\,,Z_2=\hat\beta\sqrt{s_{xx}}$ and $RSS=\sum_{i=3}^n Z_i^2$ where the $Z_i$'s are all independently distributed normal random variables. This shows the independence of $\hat\beta$ and $RSS$. Now $\hat\alpha=\overline y-\hat\beta\overline x$ is a function of $Z_1,Z_2$ only which is therefore also independent of $RSS$. A general proof is shown here: stats.stackexchange.com/q/173396/119261. $\endgroup$– StubbornAtomMar 27, 2021 at 19:20
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
It is easiest if you take a geometric approach. The coefficient vector and the residual are independent if they are orthogonal.
$$ (y - \hat{y})^T \beta $$
If you replace $\beta$ by the projection matrix and work through the algebra, what do you get?
answered Oct 8, 2020 at 4:00
-
$\begingroup$ I understand that the coefficient vector and the residual are independent. I am just wondering how do I prove the independence for the sum of square of residuals. Besides, what do you mean by replace $\beta$ by the projection matrix? $\endgroup$– Van TomOct 8, 2020 at 4:56