2
$\begingroup$

In looking over TooTone's answer in Derive Variance of regression coefficient in simple linear regression, there's a step in line 3 below where $(\beta_0 + \beta_1x_i + u_i )$ is simplified to $u_i$ on the basis that only $u_i$ is a random variable. I'm having trouble understanding why the $\beta_0 + \beta_1x_i$ terms can be ignored as a result of this? Is anyone able to offer more detail on either a reason or an intermediate step that I'm not following?

\begin{align} \text{Var}(\hat{\beta_1}) & = \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \right) \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{substituting in the above} \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})u_i}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{noting only $u_i$ is a random variable} \\ &= \frac{\sum_i (x_i - \bar{x})^2\text{Var}(u_i)}{\left(\sum_i (x_i - \bar{x})^2\right)^2} , \;\;\;\text{independence of } u_i \text{ and, Var}(kX)=k^2\text{Var}(X) \\ &= \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2} \\ \end{align}

$\endgroup$
4
  • $\begingroup$ When $U$ is random and $x$ is a constant, $\operatorname{Var}(xU)=x^2\operatorname{Var}(U).$ See stats.stackexchange.com/…. $\endgroup$
    – whuber
    Jun 2, 2019 at 19:06
  • $\begingroup$ I guess I'm having trouble seeing how $\sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )$ meets $Var(xU)$? $\endgroup$ Jun 2, 2019 at 19:35
  • $\begingroup$ Check out the search I gave, then, which links to posts that express the basic properties of variance. $\endgroup$
    – whuber
    Jun 2, 2019 at 19:49
  • $\begingroup$ I ended up turning to wikipedia, but I guess the piece I was missing was that $Var(X + a) = Var(X)$. $\endgroup$ Jun 2, 2019 at 20:13

2 Answers 2

2
$\begingroup$

If $X$ and $Y$ are independent random variables, then $$ \textrm{Var}(X+Y)=\textrm{Var} X+\textrm{Var} Y.\qquad (1) $$ This is because $$ \mathbb E\left[(X+Y)^2\right]=\mathbb E[X^2]+2\mathbb E[XY]+\mathbb E[Y^2]\overset{indep!}=\mathbb E[X^2]+2\mathbb EX\mathbb EY+\mathbb E[Y^2]=\left(\mathbb EX+\mathbb EY\right)^2. $$ Similarly if $a$ is deterministic and $X$ is random, then $$\textrm{Var}(aX)=a^2\textrm{Var}X,\qquad (2)$$ since $$ \mathbb E[(aX)^2]=\mathbb E[a^2X^2]=a^2\mathbb E[X^2]. $$ Finally, if $a$ is deterministic and $X$ is random, then $$\textrm{Var}(a+X)=\textrm{Var}X,\qquad (3)$$ since $$ \mathbb E\left[a+X-\mathbb E(a+X)\right]^2=\mathbb E\left[X-\mathbb EX\right]^2. $$

We can use these three properties to answer your question. You have a sequence of independent random variables $(u_i)$ and sequences of constants $(a_i),(b_i)$ where $$a_i=\frac{x_i-\overline{x_i}}{\sum_i(x_i-\overline{x_i})^2},\qquad b_i=\beta_0+\beta_1 x_i$$ and you wish to simplify the expression $$ \textrm{Var}\left(\sum_i a_i(b_i+u_i)\right). $$ Using the third property, $$ \textrm{Var}\left(\sum_i a_i(b_i+u_i)\right)=\textrm{Var}\left(\sum_i a_iu_i\right). $$ Using the first property and independence, $$ \textrm{Var}\left(\sum_i a_iu_i\right)=\sum_i \textrm{Var}(a_iu_i). $$ Using the second property, $$ \sum_i \textrm{Var}(a_iu_i)=\sum_i a_i^2\textrm{Var}u_i. $$ Altogether, this fills in the missing reasoning.

$\endgroup$
1
$\begingroup$

The property used here is that $\mathbb{Var}(\text{const}+X) = \mathbb{Var}(X)$. If you expand the terms out formally, from the second step to the third, you get:

$$\begin{equation} \begin{aligned} \mathbb{Var} (\hat{\beta}_1) &= \mathbb{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1 x_i + u_i )}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1 x_i) }{\sum_i (x_i - \bar{x})^2} + \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left( \text{const} + \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left( \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right). \\[6pt] \end{aligned} \end{equation}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.