I have a question on the distribution of betas in a multiple linear regression scheme

The estimated parameter vector is $\hat{\beta}=(X^′X)^{−1}X^′y$ where $X = [1 \; \;x]$ is the $n \times 2$ data matrix.

Substitute $X \beta + \epsilon$ for y.

Calculate $\text{var}(\hat{\beta})=\text{var}[(\beta+(X^′X)^{−1}X^′\epsilon)]$

Using this relation, how do we get that


where $e=y-X\hat{\beta}$?

  • $\begingroup$ In particular, I read that the degrees of freedom are N-p-1 where p is the number of parameters, how do we get this? $\endgroup$ – bgbgh Sep 25 '11 at 22:11
  • 1
    $\begingroup$ Regarding degrees of freedom, look at this CV Q&A or this nice explanation referred to in one of the answers. $\endgroup$ – Karl Sep 25 '11 at 23:41
  • $\begingroup$ Re-write your variance as $\hat{\text{var}}[\hat{\beta}]=[(X^′X)^{−1}/(N-p-1)]\sum_i(e_i^2)=\hat{\sigma}^2(X^′X)^{−1}$. So it seems like your question is more related to why do we estimate $\sigma^2$ as $\frac{1}{N-p-1}\sum_ie_i^2$? $\endgroup$ – probabilityislogic Sep 26 '11 at 21:44

It doesn't help much to substitute $X\beta + \epsilon$ for $y$.

The key is that $(X^'X)^{-1} X^'$ is fixed, and $\text{var}(y|X) = \sigma^2 \; I$, and we use the fact that $\text{var}(Az) = A\; \text{var}(z) \; A^'$, so:

$$\text{var}[\hat{\beta}] = \text{var}[(X^'X)^{-1}X^'y] = (X^'X)^{-1}X^'\text{var}[y]X(X^'X)^{-1} = \sigma^2(X^'X)^{-1}$$

and then we estimate $\sigma^2$ by $\sum_i e_i^2/(n-2)$.

[$n-p-1$ reduces to $n-2$ when the number of non-intercept columns is $p=1$.]

| cite | improve this answer | |
  • $\begingroup$ Thanks a lot for your reply Prof. Broman. Two follow up questions, please forgive my ignorance 1)How can we estimate sigma, by just calculating the error variance 2) What it the notion of degrees of freedom, why is is n-p-1 $\endgroup$ – bgbgh Sep 26 '11 at 0:43
  • $\begingroup$ Regarding degrees of freedom, look at the links in my comment under your question. Regarding estimating $\sigma^2$, what don't you understand? $\endgroup$ – Karl Sep 26 '11 at 1:12
  • $\begingroup$ Sure, so the variance of y can be calculated from the data set right, as sum(y-ybar)^2/n-1. $\endgroup$ – bgbgh Sep 26 '11 at 1:48
  • $\begingroup$ where ybar is the mean of the values of y. How do the error give the variance estimate. In general that leads to a more theoretical argument. When we postulate a linear model, say y = Xbeta + e (1), lets say that here beta are the population measure for the coefficients and hence their variances are all zero. Using OLS we end up estimating betas, say the estimators are beta'. $\endgroup$ – bgbgh Sep 26 '11 at 1:48
  • $\begingroup$ So now we have the equation y = Xbeta' + e' (2). Note that this e' is different from e, above. Is this reasoning sound?.In equation (1) V(y) = V(e), since V(Xbeta) = 0, where V is variance. $\endgroup$ – bgbgh Sep 26 '11 at 1:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.