Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
We have a simple linear model $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$ with the usual assumptions, $i \in \{1, \cdots, n\}$. Let $\hat{\beta_1}$ and $\hat{\sigma}^2$ be the least-square estimators for $\beta_1$ and the variance of the error terms respectively. In order to prove that $$\frac{\hat{\beta_1}}{\hat{\sigma}/\sqrt{n}S_x} \sim T_{n-2}$$ where $S_x=\frac{1}{n}\sum_i{(x_i-\bar{x})^2}$, we have to show that $\hat{\beta_1}$ and $\hat{\sigma}$ are independent.
How can we do this?
This is tantamount to showing that $\widehat{\boldsymbol{\beta}}$, the vector of estimates, is independent of the residual vector $\mathbf{e}$. I trust that you are familiar with the matrix notation of the model, it makes the proof quite short.
Recall that the OLS estimator is given by $\left( \mathbf{X}^{T}\mathbf{X} \right)^{-1}\mathbf{X}^{T}\mathbf{Y}$ and the vector of residuals by $\left(\mathbf{I}-\mathbf{H} \right)\mathbf{Y}$, where $\mathbf{H}$ is the projection matrix given by $\mathbf{X}\left(\mathbf{X}^{T}\mathbf{X} \right)^{-1}\mathbf{X}^{T}$. Assuming that $\mathbf{Y}$ is multivariate normal, which is the assumption you need in order to construct finite-sample tests and confidence intervals, what we want to do is take advantage of the fact that linear combinations of multivariate normal variables are also multivariate normal. Hence we rewrite these two as follows
$$\begin{bmatrix} \widehat{\boldsymbol{\beta}} \\ \mathbf{e} \end{bmatrix}=\begin{bmatrix} \left( \mathbf{X}^{T}\mathbf{X} \right)^{-1}\mathbf{X}^{T} \\ \mathbf{I}-\mathbf{H} \end{bmatrix} \mathbf{Y}$$
And now we need to remember that if $\mathbf{X}\sim N_p \left(\boldsymbol{\mu}, \boldsymbol{\Sigma} \right)$, then $\mathbf{AX}\sim N_p \left(\mathbf{A}\boldsymbol{\mu}, \mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^{T} \right)$ (the distribution is closed under affine transformations). We are mainly interested in the new covariance matrix as it being diagonal will indicate that the variables are independent and so we focus on that. It is easy to show-and I leave the details to you- that the covariance matrix is of the form
$$ \begin{bmatrix} \sigma^2 \left(\mathbf{X}^{T}\mathbf{X} \right)^{-1} & \mathbf{0} \\ \mathbf{0} & \sigma^2 \left(\mathbf{I}-\mathbf{H} \right) \end{bmatrix} $$
and so we may conclude that the random variables are independent. And since $\widehat{\boldsymbol{\beta}}$ is independent of $\mathbf{e}$, it is also independent of the Mean Squared Error $\frac{\mathbf{e}^{T}\mathbf{e}}{n-k}$, as we wanted to show.
Note that in general lack of correlation does not imply independence, this is a special property of the multivariate normal distribution and undoubtedly one of the reasons it is loved so much by statisticians.
Hope this helps.
