I am reading a text, "Probability and Statistics" by Devore. I am looking at 2 items on page 740: the expected value and variance of the estimation of $\beta_1$, which is the slope parameter in the linear regression $Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$. $\epsilon_i$ is a Gaussian($\mu = 0, variance=\sigma^2$) random variable and the $\epsilon_i$ are independent.

The estimate of $\beta_1$ can be expressed as: $\hat{\beta_1} = \frac{\sum (x_i - \bar{x}) (Y_i - \bar{Y})}{\sum(x_i-\bar{x})^2} = \frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}$, where $S_{xx} = \sum (x_i - \bar{x})^2$. So, my question is: how do I derive $E(\hat{\beta_1})$ and $Var(\hat{\beta_1})$? The book has already given the results: $E(\hat{\beta_1}) = \beta_1$ and $Var(\hat{\beta_1}) = \frac{\sigma^2}{S_xx}$.

My work in the derivation: $E\left(\frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}\right) = E\left(\frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon)}{S_{xx}}\right) = E\left(\frac{\sum (x_i - \bar{x})\beta_1 x_i}{S_{xx}}\right)$, since $\sum(x_i - \bar{x})c = 0$ and $E(c\epsilon) = 0$. But I am stuck.

Also, $Var\left(\frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon)}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})\epsilon}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})}{S_{xx}}\right) \sigma^2$, but I am stuck.

  • $\begingroup$ My comment on June 22, 2011 in user whuber's answer should include the subscript $i$ in the $\epsilon$'s, and should make use of the fact that the error terms $\epsilon_i$ are independent. $\endgroup$ – jrand Mar 20 '13 at 17:57
  • $\begingroup$ $\mathrm{Var}(\hat{\beta_1}) = \mathrm{Var}\left(\sum\frac{(x_i - \bar{x})y_i}{S_{xx}}\right) = \mathrm{Var}\left(\sum{\frac{(x_i-\bar{x})\epsilon_i}{S_{xx}}}\right) = \mathrm{Var}\left(\frac{(x_1 - \bar{x})\epsilon_1}{S_{xx}} + \frac{(x_2 - \bar{x})\epsilon_2}{S_{xx}} + \ldots + \frac{(x_n - \bar{x})\epsilon_n}{S_{xx}}\right) = \frac{(x_1 - \bar{x})^2 \sigma^2}{(S_{xx})^2} + \frac{(x_2 - \bar{x})^2 \sigma^2}{(S_{xx})^2} + \ldots + \frac{(x_n - \bar{x})^2 \sigma^2}{(S_{xx})^2} = \sigma^2 \left[\sum{\frac{(x_i-\bar{x})^2}{(S_{xx})^2}}\right] = \frac{\sigma^2}{S_{xx}}$ $\endgroup$ – jrand Mar 20 '13 at 17:57
  • $\begingroup$ The standard "answer" is an underestimate, it ignores the variance of S_{xx}. $\endgroup$ – climbert8 Oct 7 '16 at 4:37
  • 1
    $\begingroup$ In the situation being asked about, $X$ is being conditioned on, so it's treated as fixed rather than random $\endgroup$ – Glen_b Oct 7 '16 at 7:05
  1. $E\left(\frac{\sum (x_i - \bar{x})\beta_1 x_i}{S_{xx}}\right)$ = $\frac{\sum (x_i - \bar{x})\beta_1 x_i}{S_{xx}}$ because everything is constant. The rest is just algebra. Evidently you need to show $\sum (x_i - \bar{x}) x_i = S_{xx}$. Looking at the definition of $S_{xx}$ and comparing the two sides leads one to suspect $\sum(x_i - \bar{x}) \bar{x} = 0$. This follows easily from the definition of $\bar{x}$.

  2. $Var\left(\frac{\sum (x_i - \bar{x})\epsilon}{S_{xx}}\right)$ = $\sum \left[\frac{(x_i - \bar{x})^2}{S_{xx}^2}\sigma^2\right] $. It simplifies, using the definition of $S_{xx}$, to the desired result.

  • 2
    $\begingroup$ For the 2nd point, the variance, the equation should be: $Var\left(\frac{\sum (x_i - \bar{x})\epsilon}{S_{xx}}\right) = Var\left( \frac{(x_1 - \bar{x})\epsilon + (x_2 - \bar{x})\epsilon + \ldots + (x_n - \bar{x})\epsilon}{S_{xx}} \right) = \left( \frac{ \sum_i^{n} (x_i - \bar{x})^2} {S_{xx}^2} \right) \times \sigma^2 = \frac{S_{xx}}{S_{xx}^2} \times \sigma^2 = \frac{\sigma^2}{S_{xx}}$ $\endgroup$ – jrand Jun 22 '11 at 16:09
  • 1
    $\begingroup$ @jrand Yes, that's exactly what I wrote (although your first equality doesn't accomplish anything: it's just a more laborious way to write the summation). The whole point--and the thing worth remembering--is that when $\varepsilon$ is a random variable with a variance and $\lambda$ is constant, $Var(\lambda \varepsilon)$ = $\lambda^2 Var(\varepsilon)$. $\endgroup$ – whuber Jun 22 '11 at 16:14
  • $\begingroup$ Unless I am getting the notation wrong, this is a false statement: $\sum_i^n (x_i)^2 = \left( \sum_i^n x_i \right)^2$. The quantity on the left hand side is the sum of squares, and the other is the square of the sum. $\endgroup$ – jrand Jun 22 '11 at 17:34
  • $\begingroup$ @jrand You are correct: there is a typographical error in my reply. Thank you for pointing it out. I have reformatted it to fix the error and make the logic clearer. $\endgroup$ – whuber Jun 22 '11 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.