0
$\begingroup$

Recap of required theory

Consider the following regression:

$$y_i = \alpha + \beta x_i + u_i \tag{1}$$

where $y_i$ are iid and $x_i$ are deterministic (i.e. fixed).

We know that the OLS estimator $\hat{\beta}$ is:

$$ \hat{\beta} = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

Assume that the Gauss-Markov conditions are fulfilled. Then, the OLS estimator has the minimum variance of any linear (in $y_i$) unbiased estimator.

Setup for my problem

Consider the following class of linear estimators:

$$ \tilde{\beta} = \frac{\sum_{i=1}^n a_i y_i}{\sum_{i=1}^n a_i x_i} \text{ where } \sum_{i=1}^n a_i = 0 \tag{2}$$

Notice that if we set $a_i = (x_i - \bar{x})$, we get the OLS estimator.

This class of estimators is unbiased; we can show this by substituting (1) into (2), leading to:

$$ \tilde{\beta} = \beta + \frac{\sum_{i=1}^n a_i u_i}{\sum_{i=1}^n a_i x_i} \tag{3}$$

Given that the error terms $u_i$ are exogenous, taking expectations through (3) completes the proof that this class of estimators is indeed unbiased.

Why have I bothered to show that this class of estimators is unbiased? Because if it is, then by the Gauss-Markov theorem:

Setting $a_i = (x_i - \bar{x})$ minimizes the variance of $\tilde{\beta}$.

This is the result I want to prove.


My problem: Minimizing the variance of $\tilde{\beta}$

Assuming homoskedasticity i.e $V(u_i) = \sigma^2$, we can use (3) to show that:

$$ V(\tilde{\beta}) = \frac{\sigma^2 \sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i x_i\right]^2} \tag{4}$$

This is a one-variable optimization problem, since $x_i$ are fixed. Differentiating wrt $a_i$ using the quotient rule yields the first order condition:

$$ \frac{dV}{da} = \frac{2\sigma^2\left[\sum_{i=1}^n a_i x_i\right]^2\left[\sum_{i=1}^n a_i\right] - 2\sigma^2\left[\sum_{i=1}^n a_i x_i^2\right]\left[\sum_{i=1}^n a_i^2\right] }{\left[\sum_{i=1}^n a_i x_i\right]^4} = 0 \tag{5}$$

This simplifies down to:

$$ \left[\sum_{i=1}^n a_i x_i\right]^2\left[\sum_{i=1}^n a_i\right] = \left[\sum_{i=1}^n a_i x_i^2\right]\left[\sum_{i=1}^n a_i^2\right] \tag{6} $$

Using that $\sum_{i=1}^n a_i = 0$, this becomes:

$$\left[\sum_{i=1}^n a_i x_i^2\right]\left[\sum_{i=1}^n a_i^2\right] = 0 \tag{7}$$

...and this is where I get stuck.

How can I solve (7) to show that:

$$a_i = x_i - \bar{x}$$

I realize that this is simply a question of algebraic manipulation (I think I've derived everything up to (7) correctly); I just can't see how to simplify it.

$\endgroup$
  • 1
    $\begingroup$ At first, $\tilde \beta$ is not unbiased estimate because $E(\tilde \beta) = \alpha \sum a_i +\beta$. $\endgroup$ – user158565 Nov 3 '18 at 2:45
  • $\begingroup$ @a_statistician Thanks. I've added a missing detail to (2) which justifies the unbiasedness. $\endgroup$ – Thevesh Theva Nov 3 '18 at 2:50
  • $\begingroup$ eq. (5), should be partial derivative to $a_1, a_2,...,a_n$, so you have $n$ partial derivatives, and set them to 0, get the solution of$a_1,a_n$? I think so. $\endgroup$ – user158565 Nov 3 '18 at 3:34
  • $\begingroup$ We're dealing with well-behaved function, so the derivative of the sum is the sum of the derivatives. I don't understand your comment beyond that, I'm afraid. $\endgroup$ – Thevesh Theva Nov 3 '18 at 3:53
  • $\begingroup$ I think it should be: $\frac {\partial V}{\partial a_1} = 0$, ..., : $\frac {\partial V}{\partial a_n} = 0$ $\endgroup$ – user158565 Nov 3 '18 at 5:18
1
$\begingroup$

Because $\sum_{i=1}^n a_i = 0$, $\sum_{i=1}^n a_i x_i =\sum_{i=1}^n a_i x_i -\bar x \sum_{i=1}^n a_i = \sum_{i=1}^n a_i (x_i-\bar x)$. So (4) can be written as $$V(\tilde{\beta}) = \frac{\sigma^2 \sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i x_i\right]^2}=\frac{\sigma^2 \sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i (x_i - \bar x)\right]^2}$$

According to Cauchy-Schwarz Inequality, we have $$\left[\sum_{i=1}^n a_i (x_i - \bar x)\right]^2 \le \sum_{i=1}^n (x_i-\bar x)^2\sum_{i=1}^n a_i^2$$ Divided by $\left[\sum_{i=1}^n a_i (x_i - \bar x)\right]^2 \sum_{i=1}^n (x_i-\bar x)^2$ on both sides, we get: $$\frac{\sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i (x_i-\bar x)\right]^2} \ge \frac 1 {\sum_{i=1}^n (x_i-\bar x)^2} = \frac {\sum_{i=1}^n (x_i-\bar x)^2}{\left(\sum_{i=1}^n (x_i-\bar x)^2\right)^2}$$ So $V(\hat \beta) \le V(\tilde \beta)$

$\endgroup$
  • $\begingroup$ very nice. another sneakier way is to argue that $\hat{\beta}$ is the MLE and MLE's minimize the residual variance. ( the sum of (yactual - predicted ). by their definition. i.e: maximum likelihood, in the case of the normal distribution, minimizes the variance of $\epsilon$ and the variance of $\beta$ is a function of the variance of $\epsilon$. I like your way better. $\endgroup$ – mlofton Nov 3 '18 at 5:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.