2
$\begingroup$

In looking over TooTone's answer in Derive Variance of regression coefficient in simple linear regression, there's a step in line 3 below where $(\beta_0 + \beta_1x_i + u_i )$ is simplified to $u_i$ on the basis that only $u_i$ is a random variable. I'm having trouble understanding why the $\beta_0 + \beta_1x_i$ terms can be ignored as a result of this? Is anyone able to offer more detail on either a reason or an intermediate step that I'm not following?

\begin{align} \text{Var}(\hat{\beta_1}) & = \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \right) \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{substituting in the above} \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})u_i}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{noting only $u_i$ is a random variable} \\ &= \frac{\sum_i (x_i - \bar{x})^2\text{Var}(u_i)}{\left(\sum_i (x_i - \bar{x})^2\right)^2} , \;\;\;\text{independence of } u_i \text{ and, Var}(kX)=k^2\text{Var}(X) \\ &= \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2} \\ \end{align}

$\endgroup$
  • $\begingroup$ When $U$ is random and $x$ is a constant, $\operatorname{Var}(xU)=x^2\operatorname{Var}(U).$ See stats.stackexchange.com/…. $\endgroup$ – whuber Jun 2 at 19:06
  • $\begingroup$ I guess I'm having trouble seeing how $\sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )$ meets $Var(xU)$? $\endgroup$ – Julian Drago Jun 2 at 19:35
  • $\begingroup$ Check out the search I gave, then, which links to posts that express the basic properties of variance. $\endgroup$ – whuber Jun 2 at 19:49
  • $\begingroup$ I ended up turning to wikipedia, but I guess the piece I was missing was that $Var(X + a) = Var(X)$. $\endgroup$ – Julian Drago Jun 2 at 20:13
1
$\begingroup$

If $X$ and $Y$ are independent random variables, then $$ \textrm{Var}(X+Y)=\textrm{Var} X+\textrm{Var} Y.\qquad (1) $$ This is because $$ \mathbb E\left[(X+Y)^2\right]=\mathbb E[X^2]+2\mathbb E[XY]+\mathbb E[Y^2]\overset{indep!}=\mathbb E[X^2]+2\mathbb EX\mathbb EY+\mathbb E[Y^2]=\left(\mathbb EX+\mathbb EY\right)^2. $$ Similarly if $a$ is deterministic and $X$ is random, then $$\textrm{Var}(aX)=a^2\textrm{Var}X,\qquad (2)$$ since $$ \mathbb E[(aX)^2]=\mathbb E[a^2X^2]=a^2\mathbb E[X^2]. $$ Finally, if $a$ is deterministic and $X$ is random, then $$\textrm{Var}(a+X)=\textrm{Var}X,\qquad (3)$$ since $$ \mathbb E\left[a+X-\mathbb E(a+X)\right]^2=\mathbb E\left[X-\mathbb EX\right]^2. $$

We can use these three properties to answer your question. You have a sequence of independent random variables $(u_i)$ and sequences of constants $(a_i),(b_i)$ where $$a_i=\frac{x_i-\overline{x_i}}{\sum_i(x_i-\overline{x_i})^2},\qquad b_i=\beta_0+\beta_1 x_i$$ and you wish to simplify the expression $$ \textrm{Var}\left(\sum_i a_i(b_i+u_i)\right). $$ Using the third property, $$ \textrm{Var}\left(\sum_i a_i(b_i+u_i)\right)=\textrm{Var}\left(\sum_i a_iu_i\right). $$ Using the first property and independence, $$ \textrm{Var}\left(\sum_i a_iu_i\right)=\sum_i \textrm{Var}(a_iu_i). $$ Using the second property, $$ \sum_i \textrm{Var}(a_iu_i)=\sum_i a_i^2\textrm{Var}u_i. $$ Altogether, this fills in the missing reasoning.

$\endgroup$
1
$\begingroup$

The property used here is that $\mathbb{Var}(\text{const}+X) = \mathbb{Var}(X)$. If you expand the terms out formally, from the second step to the third, you get:

$$\begin{equation} \begin{aligned} \mathbb{Var} (\hat{\beta}_1) &= \mathbb{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1 x_i + u_i )}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1 x_i) }{\sum_i (x_i - \bar{x})^2} + \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left( \text{const} + \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left( \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right). \\[6pt] \end{aligned} \end{equation}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.