3
$\begingroup$

Suppose I estimate the model

$$y_i=\beta_0+\beta_1 x_i +\epsilon_i$$

via weighted least squares where $x_i$ and $\epsilon_i$ are independent, but I use weights $w_i$ that may be correlated with both $x_i$ and $\epsilon_i$. Is $\hat\beta_1$ still unbiased? I am having trouble trying to prove/disprove this.

The bias takes the form

$$ \frac{\sum_i w_{i}\left(x_{i}-\bar{x}\right)\left(\epsilon_{i}-\bar{\epsilon}\right)}{\sum_i w_{i}\left(x_{i}-\bar{x}\right)^{2}} $$

where $\bar x,\bar \epsilon$ are the weighted averages of $x,\epsilon$, but have not made any more progress.

Thanks!

Edit: I guess you can recast this as the OLS

$$\sqrt{w_i}y_i=\beta_0\sqrt{w_i}+\beta_1\sqrt{w_i}x_i+\sqrt{w_i}\epsilon_i$$

and the question becomes what is the relationship between $\sqrt{w_i}x_i$ and $\sqrt{w_i}\epsilon_i$?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

No, your coefficient estimators are not necessarily unbiased: Since you have specified that the weights may be correlated with the error terms, this means that they are random variables in your analysis. By allowing the weights to be related to the error terms, this allows models in which data points may be given high/low weighting based on the error term, conditional on the explanatory variable. This allows for the creation of biased estimators by favouring data points that have certain error properties that cause a higher/lower coefficient estimate.

To see this, let's consider a special case where $W_i \in \{ 0,1 \}$ so that the weights are either zero or one. This is tantamount to using OLS estimation, but with only a subset of the data points, where that subset can depend on both the explanatory variable $x_i$ and the error term $\epsilon_i$. Clearly, using this combination of values, it is easy for us to specify a model where we only include data points that over-estimate the coefficients. For example, we could take $W_i = \mathbb{I}(\text{sgn} (\epsilon_i) = \text{sgn}(x_i - \bar{x}))$, yielding correlation between the weights and the errors/explanatory variables. In this case the coefficient estimator is an OLS estimator, except that it only uses data points that have positive error when $x_i > \bar{x}$ and negative error when $x_i > \bar{x}$. This would clearly lead to systematic overestimation of the slope coefficient.

To get a better understanding of what is going on here, it is worth deriving an explicit equation for the bias of the WLS estimator. Letting $W$ be the weight matrix containing the weights of interest, we have:

$$\begin{equation} \begin{aligned} \text{Bias}[\hat{\beta}_W] = \mathbb{E}[\hat{\beta}_W] - \beta &= \mathbb{E}[ (x^\text{T} W x)^{-1} (x^\text{T} W Y) ] - \beta \\[6pt] &= \mathbb{E}[ (x^\text{T} W x)^{-1} x^\text{T} W \epsilon] \\[6pt] &= \mathbb{Cov}[ (x^\text{T} W x)^{-1} x^\text{T} W, \epsilon]. \end{aligned} \end{equation}$$

In the standard case where $W \bot \epsilon$ this reduces to zero (i.e., unbiasedness), but in the case where these can be correlated, you can have bias.

Improve this answer
$\endgroup$
5
  • $\begingroup$ what if $w_i$ is correlated with $\epsilon_i$ but not $x_i$? I could see this biasing $\hat\beta_0$, but would this also bias $\hat\beta_1$? Similarly, if it is correlated with $x_i$ but not $\epsilon_i$? Thank you! $\endgroup$
    – rkatzwer
    Mar 15, 2018 at 22:14
  • $\begingroup$ This can still give bias. $\endgroup$
    – Ben
    Mar 15, 2018 at 23:25
  • $\begingroup$ but in your second part, where you derive the bias, it depends on the covariance of $(x'Wx)^-1x'W$ and $\epsilon$. If $W$ is independent of $\epsilon$ you should still be unbiased, right? $\endgroup$
    – rkatzwer
    Mar 16, 2018 at 0:23
  • $\begingroup$ Yes. That's exactly what the last sentence of the answer says. ;) $\endgroup$
    – Ben
    Mar 16, 2018 at 0:40
  • $\begingroup$ Ah! You're right, I missed that last line, thanks! $\endgroup$
    – rkatzwer
    Mar 16, 2018 at 0:45
0
$\begingroup$

When the mean model is correct, then the least squares estimator is still unbiased for any arbitrary weighting mechanism. The consequence of using the wrong weights is that the ordinary least squares is inefficient. To prove WLS is unbiased you merely need to show that:

$$\begin{eqnarray} E(\hat{\beta_W}) &=& E(\left(\mathbf{X}\mathbf{W} \mathbf{X} \right)^{-1} \mathbf{X}\mathbf{W} Y \\ &=& \left(\mathbf{X}\mathbf{W} \mathbf{X} \right)^{-1} \mathbf{X}\mathbf{W} E(Y) \\ &=& \left(\mathbf{X}\mathbf{W} \mathbf{X} \right)^{-1} \mathbf{X}\mathbf{W} \mathbf{X} \beta \\ &=& \beta\\ \end{eqnarray}$$

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for the reply, but I am having trouble squaring this with Ben's answer above which makes sense to me. Going from line one two two, where you move the expectation from the $E[(X'WX)^{-1}XWY]$ to $E[Y]$, what assumptions of X, W do you need to make to do that? $\endgroup$
    – rkatzwer
    Mar 15, 2018 at 21:54
  • $\begingroup$ If $W$ can depend on $\epsilon$ then you can't take it out of the expectation operator, so this reasoning seems to me to be invalid. $\endgroup$
    – Ben
    Mar 15, 2018 at 21:56
  • $\begingroup$ @Ben the weights are not a random variable. If you treat them as such, then I bet the estimates are possibly biased even if $W$ does not depend on errors. $\endgroup$
    – AdamO
    Mar 15, 2018 at 22:05
  • $\begingroup$ @AdamO the use case was that $W$ was potentially correlated with $x$ and $\epsilon$ $\endgroup$
    – rkatzwer
    Mar 15, 2018 at 22:11
  • $\begingroup$ @rkatzwer neither is the $x$ a random variable. But even then, I believe that applying the law of total expectation would lead to the same result $E(\hat{\beta}_W) = E(E(\hat{\beta}_W)|X,W)$ $\endgroup$
    – AdamO
    Mar 15, 2018 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.