12
$\begingroup$

I am reading a text, "Probability and Statistics" by Devore. I am looking at 2 items on page 740: the expected value and variance of the estimation of $\beta_1$, which is the slope parameter in the linear regression $Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$. $\epsilon_i$ is a Gaussian($\mu = 0, variance=\sigma^2$) random variable and the $\epsilon_i$ are independent.

The estimate of $\beta_1$ can be expressed as: $\hat{\beta_1} = \frac{\sum (x_i - \bar{x}) (Y_i - \bar{Y})}{\sum(x_i-\bar{x})^2} = \frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}$, where $S_{xx} = \sum (x_i - \bar{x})^2$. So, my question is: how do I derive $E(\hat{\beta_1})$ and $Var(\hat{\beta_1})$? The book has already given the results: $E(\hat{\beta_1}) = \beta_1$ and $Var(\hat{\beta_1}) = \frac{\sigma^2}{S_xx}$.

My work in the derivation: $E\left(\frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}\right) = E\left(\frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon)}{S_{xx}}\right) = E\left(\frac{\sum (x_i - \bar{x})\beta_1 x_i}{S_{xx}}\right)$, since $\sum(x_i - \bar{x})c = 0$ and $E(c\epsilon) = 0$. But I am stuck.

Also, $Var\left(\frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon)}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})\epsilon}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})}{S_{xx}}\right) \sigma^2$, but I am stuck.

Improve this question
$\endgroup$
5
  • $\begingroup$ My comment on June 22, 2011 in user whuber's answer should include the subscript $i$ in the $\epsilon$'s, and should make use of the fact that the error terms $\epsilon_i$ are independent. $\endgroup$
    – jrand
    Mar 20, 2013 at 17:57
  • $\begingroup$ $\mathrm{Var}(\hat{\beta_1}) = \mathrm{Var}\left(\sum\frac{(x_i - \bar{x})y_i}{S_{xx}}\right) = \mathrm{Var}\left(\sum{\frac{(x_i-\bar{x})\epsilon_i}{S_{xx}}}\right) = \mathrm{Var}\left(\frac{(x_1 - \bar{x})\epsilon_1}{S_{xx}} + \frac{(x_2 - \bar{x})\epsilon_2}{S_{xx}} + \ldots + \frac{(x_n - \bar{x})\epsilon_n}{S_{xx}}\right) = \frac{(x_1 - \bar{x})^2 \sigma^2}{(S_{xx})^2} + \frac{(x_2 - \bar{x})^2 \sigma^2}{(S_{xx})^2} + \ldots + \frac{(x_n - \bar{x})^2 \sigma^2}{(S_{xx})^2} = \sigma^2 \left[\sum{\frac{(x_i-\bar{x})^2}{(S_{xx})^2}}\right] = \frac{\sigma^2}{S_{xx}}$ $\endgroup$
    – jrand
    Mar 20, 2013 at 17:57
  • $\begingroup$ The standard "answer" is an underestimate, it ignores the variance of S_{xx}. $\endgroup$
    – climbert8
    Oct 7, 2016 at 4:37
  • 1
    $\begingroup$ In the situation being asked about, $X$ is being conditioned on, so it's treated as fixed rather than random $\endgroup$
    – Glen_b
    Oct 7, 2016 at 7:05
  • $\begingroup$ How can you get $Y_i = Y_i - \bar{Y}$ $\endgroup$
    – martinrhan
    Oct 21 at 8:31

1 Answer 1

Reset to default
9
$\begingroup$

  1. $E\left(\frac{\sum (x_i - \bar{x})\beta_1 x_i}{S_{xx}}\right)$ = $\frac{\sum (x_i - \bar{x})\beta_1 x_i}{S_{xx}}$ because everything is constant. The rest is just algebra. Evidently you need to show $\sum (x_i - \bar{x}) x_i = S_{xx}$. Looking at the definition of $S_{xx}$ and comparing the two sides leads one to suspect $\sum(x_i - \bar{x}) \bar{x} = 0$. This follows easily from the definition of $\bar{x}$.

  2. $Var\left(\frac{\sum (x_i - \bar{x})\epsilon}{S_{xx}}\right)$ = $\sum \left[\frac{(x_i - \bar{x})^2}{S_{xx}^2}\sigma^2\right] $. It simplifies, using the definition of $S_{xx}$, to the desired result.

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ For the 2nd point, the variance, the equation should be: $Var\left(\frac{\sum (x_i - \bar{x})\epsilon}{S_{xx}}\right) = Var\left( \frac{(x_1 - \bar{x})\epsilon + (x_2 - \bar{x})\epsilon + \ldots + (x_n - \bar{x})\epsilon}{S_{xx}} \right) = \left( \frac{ \sum_i^{n} (x_i - \bar{x})^2} {S_{xx}^2} \right) \times \sigma^2 = \frac{S_{xx}}{S_{xx}^2} \times \sigma^2 = \frac{\sigma^2}{S_{xx}}$ $\endgroup$
    – jrand
    Jun 22, 2011 at 16:09
  • 1
    $\begingroup$ @jrand Yes, that's exactly what I wrote (although your first equality doesn't accomplish anything: it's just a more laborious way to write the summation). The whole point--and the thing worth remembering--is that when $\varepsilon$ is a random variable with a variance and $\lambda$ is constant, $Var(\lambda \varepsilon)$ = $\lambda^2 Var(\varepsilon)$. $\endgroup$
    – whuber ♦
    Jun 22, 2011 at 16:14
  • $\begingroup$ Unless I am getting the notation wrong, this is a false statement: $\sum_i^n (x_i)^2 = \left( \sum_i^n x_i \right)^2$. The quantity on the left hand side is the sum of squares, and the other is the square of the sum. $\endgroup$
    – jrand
    Jun 22, 2011 at 17:34
  • $\begingroup$ @jrand You are correct: there is a typographical error in my reply. Thank you for pointing it out. I have reformatted it to fix the error and make the logic clearer. $\endgroup$
    – whuber ♦
    Jun 22, 2011 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.