# Estimators independence in simple linear regression

We have a simple linear model $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$ with the usual assumptions, $i \in \{1, \cdots, n\}$. Let $\hat{\beta_1}$ and $\hat{\sigma}^2$ be the least-square estimators for $\beta_1$ and the variance of the error terms respectively. In order to prove that $$\frac{\hat{\beta_1}}{\hat{\sigma}/\sqrt{n}S_x} \sim T_{n-2}$$ where $S_x=\frac{1}{n}\sum_i{(x_i-\bar{x})^2}$, we have to show that $\hat{\beta_1}$ and $\hat{\sigma}$ are independent.

How can we do this?

## 1 Answer

This is tantamount to showing that $\widehat{\boldsymbol{\beta}}$, the vector of estimates, is independent of the residual vector $\mathbf{e}$. I trust that you are familiar with the matrix notation of the model, it makes the proof quite short.

Recall that the OLS estimator is given by $\left( \mathbf{X}^{T}\mathbf{X} \right)^{-1}\mathbf{X}^{T}\mathbf{Y}$ and the vector of residuals by $\left(\mathbf{I}-\mathbf{H} \right)\mathbf{Y}$, where $\mathbf{H}$ is the projection matrix given by $\mathbf{X}\left(\mathbf{X}^{T}\mathbf{X} \right)^{-1}\mathbf{X}^{T}$. Assuming that $\mathbf{Y}$ is multivariate normal, which is the assumption you need in order to construct finite-sample tests and confidence intervals, what we want to do is take advantage of the fact that linear combinations of multivariate normal variables are also multivariate normal. Hence we rewrite these two as follows

$$\begin{bmatrix} \widehat{\boldsymbol{\beta}} \\ \mathbf{e} \end{bmatrix}=\begin{bmatrix} \left( \mathbf{X}^{T}\mathbf{X} \right)^{-1}\mathbf{X}^{T} \\ \mathbf{I}-\mathbf{H} \end{bmatrix} \mathbf{Y}$$

And now we need to remember that if $\mathbf{X}\sim N_p \left(\boldsymbol{\mu}, \boldsymbol{\Sigma} \right)$, then $\mathbf{AX}\sim N_p \left(\mathbf{A}\boldsymbol{\mu}, \mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^{T} \right)$ (the distribution is closed under affine transformations). We are mainly interested in the new covariance matrix as it being diagonal will indicate that the variables are independent and so we focus on that. It is easy to show-and I leave the details to you- that the covariance matrix is of the form

$$\begin{bmatrix} \sigma^2 \left(\mathbf{X}^{T}\mathbf{X} \right)^{-1} & \mathbf{0} \\ \mathbf{0} & \sigma^2 \left(\mathbf{I}-\mathbf{H} \right) \end{bmatrix}$$

and so we may conclude that the random variables are independent. And since $\widehat{\boldsymbol{\beta}}$ is independent of $\mathbf{e}$, it is also independent of the Mean Squared Error $\frac{\mathbf{e}^{T}\mathbf{e}}{n-k}$, as we wanted to show.

Note that in general lack of correlation does not imply independence, this is a special property of the multivariate normal distribution and undoubtedly one of the reasons it is loved so much by statisticians.

Hope this helps.

• A very minor issue but I'm not used to seeing $\widehat{\mathbf{b}}$: either $\hat{\beta}$ (estimating a Greek letter for the population parameter $\beta$, sometimes written in bold) or plain $\mathbf{b}$ (using a Latin letter for an estimate from a sample). I'd lean towards $\hat\beta$ I think, because it seems to match the OP's notation. But it might well be that your notation is the standard notation somewhere! Do you know if there is a textbook that does write it as $\widehat{\mathbf{b}}$? – Silverfish Sep 21 '15 at 1:01
• @Silverfish Nope, you are right. This is the standard notation and I have adjusted it. – JohnK Sep 21 '15 at 1:06