2
$\begingroup$

Suppose we have the standard simple linear regression model: $$ Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i, $$ with $E[\varepsilon_i|X_i] = 0$ and $\text{Var}[\varepsilon_i|X_i] = \sigma^2$.

I'm trying to show that $$ E[\hat \beta_1 | \mathbf{X}] = \beta_1, $$ directly using the definition of $\hat \beta_1$, where $\mathbf{X}$ is the vector of $X_i$'s. I know there are other ways to show it but I am trying to do it this way so that I can practise working with conditional expectation. The definition of $\hat \beta_1$ is $$ \hat \beta_1 = \frac{\sum (X_i - \bar X)(Y_i - \bar Y)}{\sum (X_i - \bar X)^2}. $$

Define $$ g_i(\mathbf{X}) := \frac{X_i - \bar X}{\sum (X_i - \bar X)^2}. $$

Here's what I've done: $$ \begin{align} E[\hat \beta_1 | \mathbf{X}] & = E\bigg[\frac{\sum (X_i - \bar X)(Y_i - \bar Y)}{\sum (X_i - \bar X)^2} \bigg| \mathbf{X}\bigg] \\ & = E\bigg[\sum_i g_i(\mathbf{X})(Y_i - \bar Y) \bigg| \mathbf{X} \bigg] \\ & = \sum_i E\bigg[g_i(\mathbf{X})(Y_i - \bar Y) \bigg| \mathbf{X} \bigg] \\ & = \sum_i E[g_i(\mathbf{X})Y_i| \mathbf{X}] - \sum_i E[g_i(\mathbf{X}) \bar Y | \mathbf{X} ] \\ & = \sum_i g_i(\mathbf{X}) E[Y_i| \mathbf{X} ] - \sum_i g_i(\mathbf{X}) E[\bar Y | \mathbf{X}] \\ \end{align} $$ Because I can take the $g(\mathbf{X})$ out of the expectation it seems we can never get a constant $\beta_1$ as the final result? Where have I gone wrong? How can we show $E[\hat \beta_1 | \mathbf{X}] = \beta_1$ using this approach?

$\endgroup$
2
$\begingroup$

Following from the third line, $E[Y_i-\bar Y|\mathbf X]=(\beta_0+\beta_1X_i)-(\beta+\beta_1\bar X)=\beta_1(X_i-\bar X)$. When substituted that back, we have $$\begin{align}E[\hat \beta_1|\mathbf X]&=\sum_{i} \beta_1g_i(\mathbf X)(X_i-\bar X)=\beta_1\sum_i\frac{(X_i-\bar X)}{\sum_j (X_j-\bar X)^2}(X_i-\bar X)\\&=\beta_1\frac{\sum_i (X_i-\bar X)^2}{\sum_j (X_j-\bar X)^2}=\beta_1\end{align}$$

By the way, careful about the summation indices. $i$ in denominator expression is different from $i$ in the numerator. So, $$g_i(\mathbf X)=\frac{X_i-\bar X}{\sum_j(X_j-\bar X)^2}$$

$\endgroup$
1
$\begingroup$

You can see it as

$$ \hat{\beta}=(X^tX)^{-1}X^ty = (X^tX)^{-1}X^t(X\beta+\varepsilon) =(X^tX)^{-1}X^tX\beta + (X^tX)^{-1}X^t\varepsilon $$

So you have that

$$ \hat{\beta}=\beta + (X^tX)^{-1}X^t\varepsilon $$

And then, it is straightforward to see that

$$ \mathbb{E}(\hat\beta) = \mathbb{E}(\beta + (X^tX)^{-1}X^t\varepsilon) = \beta + (X^tX)^{-1}X^t\mathbb{E}(\varepsilon) $$

But $\mathbb{E}(\varepsilon)=0$

So $\hat\beta$ is unbiased

$\endgroup$
3
  • 2
    $\begingroup$ You appear to be treating $X$ as a constant, whereas the question concerns the case where it is a random variable. $\endgroup$ – whuber Oct 29 '20 at 17:39
  • 1
    $\begingroup$ Here I am making the common asupmtions that $X$ is a full-rank matrix of predictors, and that $\varepsilon\sim N(0, \sigma I)$. So, answering your question, I am treating X as a matrix of covariates and $\epsilon$ as a random variable. As I say, those are common assumptions. $\endgroup$ – Álvaro Méndez Civieta Oct 29 '20 at 17:47
  • 2
    $\begingroup$ Your mathematics does not reflect those assumptions, because under them $E(\hat\beta)$ should be a vector but you have obtained an expression in terms of $\beta,$ $E[\varepsilon],$ and $X$--which is a random variable. $\endgroup$ – whuber Oct 29 '20 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.