# Find the distribution of $\beta$ error in ordinary least squares

Let $\hat\beta$ be the least squares estimator of $\beta$ in the regression model $y = x \beta + \mu$ where x is an $n \times k$ matrix with rank k and $\mu \tilde ~ N(0, \sigma^2 I)$.

1) Find the non-random matrices $A$ and $B$, such that $\hat\beta$ - $\beta$ can be written in the form $A\mu$ and the sum of the squared residuals can be written as $\mu'B\mu$.

2) Show that $B$ is idempotent and that $AB=0$

3) Indicate how these results enable one to obtain the distribution of

\begin{align} \frac{ (\hat\beta - \beta)'X'X(\hat\beta - \beta) }{(Y-X\hat\beta)'(Y-X\hat\beta)}. \frac{ n-k }{k}. \end{align}

I think I can handle the second part (2), but I have a problem getting the non-random matrices and part 3

• Please explain what you have tried so far. – Christoph Hanck Oct 14 '16 at 8:41
• Please add the self-study tag as well. – Xi'an Oct 14 '16 at 8:59
• Hints: it is not true that $A$ is square as $\mu$ is $n\times 1$ and $\beta$ is $k\times 1$, so $A$ must be $k\times n$. Take the expression for the OLS estimator and substitute for $y$. – Christoph Hanck Oct 14 '16 at 10:57

## 2 Answers

(Since this is tagged as self-study, just some tips without details.)

1. Note that the solution of OLS is

$\hat{\beta} = \frac{X^Ty}{X^TX}$. (1)

In addtion, it's given that

$y = X \beta + \mu$. (2)

Now insert (2) into (1), subtract $\beta$, and show that all terms cancel out, except those multiplying $\mu$. This should give you the $A$ solving $\hat{\beta} - \beta = A\mu$.

For $B$, see the part in the Wikipedia entry discussing estimation, in particular the annihilator matrix $M$.

2. Once you solve $A$ and $B$, this should be easy. As a hint, note that in the same Wikipedia entry, the projection matrix $P$ satisfies $P^2 = P$.

3. In $\frac{ (\hat\beta - \beta)^TX^TX(\hat\beta - \beta) }{(Y-X\hat\beta)^T(Y-X\hat\beta)}$,

substitute the expression you found for $\hat\beta - \beta$, and consider how the denominator relates to the residual sum of squares. After some substitutions, use known properties of the normal distribution.

• According to the guidelines, I have calculated matrices $A$ and $B$. Check in the comment if they are correct. – Alexxio Oct 14 '16 at 20:24
• @Alexxio No, both are wrong. For the first, note that $\hat{\beta} - \beta = \frac{X^T(X\beta + \mu)}{X^TX} - \beta$, and continue from here. For the second, try to see why it is the square of the annihilator matrix. – Ami Tavory Oct 14 '16 at 22:38
• @Alexxio $\hat\beta - \beta = \frac{X^T(X\beta + \mu)}{X^TX} - \beta = \frac{X^TX\beta + X^T\mu - X^tX \beta}{X^TX} = \frac{X^T \mu}{X^T X}$. For the annihilator matrix, read the Wikipedia entry. – Ami Tavory Oct 15 '16 at 6:29
• ok so in this case, A = $\frac{x^T}{x^Tx}$ , right? – Alexxio Oct 15 '16 at 6:55
• That is correct. Please now read the wiki in the link, and see why $B$ is the square of the annihilator matrix. – Ami Tavory Oct 15 '16 at 7:07

1) The solution of OLS is

$\hat{\beta} = \frac{X^Ty}{X^TX}$. (1)

We have:

$y = X \beta + \mu$. (2)

Therefore, $\hat{\beta} = \frac{\beta + \mu}{X^TX}$. = $\frac{\beta + \mu}{X^TX}$

= $\beta - \hat\beta = x^{-1}\mu$

hence A = $x^{-1}$

2) $\beta = (y - x)^T(y - xb)$

= $(x\beta + \mu - xb)^T(x\beta + \mu - xb)$

=$\mu'\mu = \mu'B\mu$

=$B=1$ (Am not sure about this)