I have just started studying the textbook "The Elements of Statistical Learning" and I am currently reading about Linear Regression. I have a question about the distribution of the coefficients in the regression. The set up is as follows:

The linear regression model with $p$ paramaters has the form:

$$f(X)=\beta_o+\sum_{j=1}^{p} X_j \beta_j$$

Now using a least squares criterion we minimize the RSS w.r.t. $\beta$ which gives us the estimate of $\beta$ as:

$$\hat{\beta}=(X^TX)^{-1}X^T y$$

Where $X$ is the matrix of input vectors in the regression and $y$ is the vector of outputs.

We may also show that $\text{Var}(\hat{\beta})=(X^TX)^{-1}\sigma^2$, where $\sigma^2$ is the constant variance of the $y_i$.

We now asssume that the linear model is correct for the mean, and that its errors around the mean are additive Gausian, that is we make the assumption:

$$\begin{array}{ll} Y & = E(Y|X_1,\ldots, X_p)+\epsilon \\ ~& = \beta_0+ \sum_{j=1}^p X_j\beta_j+\epsilon \end{array}$$

All of the above is fine with me but I can't understand why we then have that:

$$\hat{\beta}\sim N(\beta,(X^TX)^{-1}\sigma^2)$$

I'm aware that all this material is standard and I also think that the notation/presentation is, too, so I have been brief in the explanation of my question. If this is not the case I will happily fill in more explanation.

  • 4
    $\begingroup$ from your equations, it follows that $\beta$ is a linear combination of the $y$'s, hence, as the latter are normally distributed, so are the $\beta$. $\endgroup$ – user83346 Oct 20 '15 at 15:29
  • 1
    $\begingroup$ @fcop maybe worth editing that $\hat\beta$ is normal, not $\beta$, which is assumed to be an (albeit unknown) constant $\endgroup$ – Christoph Hanck Oct 21 '15 at 3:56
  • $\begingroup$ @Christoph Hanck: you are right, but I can't edit anymore, any idea how I can solve that? Just delete an re-write? $\endgroup$ – user83346 Oct 21 '15 at 4:50
  • $\begingroup$ For reasons I do not understand, editing comments is only possible for something like 5 minutes. Indeed, just delete and repost. $\endgroup$ – Christoph Hanck Oct 21 '15 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.