For this univariate linear regression model $$y_i = \beta_0 + \beta_1x_i+\epsilon_i$$ given data set $D=\{(x_1,y_1),...,(x_n,y_n)\}$, the coefficient estimates are $$\hat\beta_1=\frac{\sum_ix_iy_i-n\bar x\bar y}{n\bar x^2-\sum_ix_i^2}$$ $$\hat\beta_0=\bar y - \hat\beta_1\bar x$$ Here is my question, according to the book and Wikipedia, the standard error of $\hat\beta_1$ is $$s_{\hat\beta_1}=\sqrt{\frac{\sum_i\hat\epsilon_i^2}{(n-2)\sum_i(x_i-\bar x)^2}}$$ How and why?

  • 1
  • @ocram, thanks, but I'm not quite capable of handling matrix stuff, I'll try. – avocado Feb 9 '14 at 9:20
  • 1
    @ocram, I've already understand how it comes. But still a question: in my post, the standard error has $(n-2)$, where according to your answer, it doesn't, why? – avocado Feb 9 '14 at 9:40
up vote 14 down vote accepted

3rd comment above: I've already understand how it comes. But still a question: in my post, the standard error has (n−2), where according to your answer, it doesn't, why?


In my post, it is found that $$ \widehat{\text{se}}(\hat{b}) = \sqrt{\frac{n \hat{\sigma}^2}{n\sum x_i^2 - (\sum x_i)^2}}. $$ The denominator can be written as $$ n \sum_i (x_i - \bar{x})^2 $$ Thus, $$ \widehat{\text{se}}(\hat{b}) = \sqrt{\frac{\hat{\sigma}^2}{\sum_i (x_i - \bar{x})^2}} $$

With $$ \hat{\sigma}^2 = \frac{1}{n-2} \sum_i \hat{\epsilon}_i^2 $$ i.e. the Mean Square Error (MSE) in the ANOVA table, we end up with your expression for $\widehat{\text{se}}(\hat{b})$. The $n-2$ term accounts for the loss of 2 degrees of freedom in the estimation of the intercept and the slope.

  • I think I get everything else expect the last part. Can you show step by step why $\hat{\sigma}^2 = \frac{1}{n-2} \sum_i \hat{\epsilon}_i^2$ ? I too know it is related to the degrees of freedom, but I do not get the math. – Mappi May 27 '16 at 15:46

another way of thinking about the n-2 df is that it's because we use 2 means to estimate the slope coefficient (the mean of Y and X)

df from Wikipedia: "...In general, the degrees of freedom of an estimate of a parameter are equal to the number of independent scores that go into the estimate minus the number of parameters used as intermediate steps in the estimation of the parameter itself ."

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.