# Showing that the minimum-variance estimator is the OLS estimator

## Recap of required theory

Consider the following regression:

$$y_i = \alpha + \beta x_i + u_i \tag{1}$$

where $$y_i$$ are iid and $$x_i$$ are deterministic (i.e. fixed).

We know that the OLS estimator $$\hat{\beta}$$ is:

$$\hat{\beta} = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

Assume that the Gauss-Markov conditions are fulfilled. Then, the OLS estimator has the minimum variance of any linear (in $$y_i$$) unbiased estimator.

## Setup for my problem

Consider the following class of linear estimators:

$$\tilde{\beta} = \frac{\sum_{i=1}^n a_i y_i}{\sum_{i=1}^n a_i x_i} \text{ where } \sum_{i=1}^n a_i = 0 \tag{2}$$

Notice that if we set $$a_i = (x_i - \bar{x})$$, we get the OLS estimator.

This class of estimators is unbiased; we can show this by substituting (1) into (2), leading to:

$$\tilde{\beta} = \beta + \frac{\sum_{i=1}^n a_i u_i}{\sum_{i=1}^n a_i x_i} \tag{3}$$

Given that the error terms $$u_i$$ are exogenous, taking expectations through (3) completes the proof that this class of estimators is indeed unbiased.

Why have I bothered to show that this class of estimators is unbiased? Because if it is, then by the Gauss-Markov theorem:

Setting $$a_i = (x_i - \bar{x})$$ minimizes the variance of $$\tilde{\beta}$$.

This is the result I want to prove.

## My problem: Minimizing the variance of $$\tilde{\beta}$$

Assuming homoskedasticity i.e $$V(u_i) = \sigma^2$$, we can use (3) to show that:

$$V(\tilde{\beta}) = \frac{\sigma^2 \sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i x_i\right]^2} \tag{4}$$

This is a one-variable optimization problem, since $$x_i$$ are fixed. Differentiating wrt $$a_i$$ using the quotient rule yields the first order condition:

$$\frac{dV}{da} = \frac{2\sigma^2\left[\sum_{i=1}^n a_i x_i\right]^2\left[\sum_{i=1}^n a_i\right] - 2\sigma^2\left[\sum_{i=1}^n a_i x_i^2\right]\left[\sum_{i=1}^n a_i^2\right] }{\left[\sum_{i=1}^n a_i x_i\right]^4} = 0 \tag{5}$$

This simplifies down to:

$$\left[\sum_{i=1}^n a_i x_i\right]^2\left[\sum_{i=1}^n a_i\right] = \left[\sum_{i=1}^n a_i x_i^2\right]\left[\sum_{i=1}^n a_i^2\right] \tag{6}$$

Using that $$\sum_{i=1}^n a_i = 0$$, this becomes:

$$\left[\sum_{i=1}^n a_i x_i^2\right]\left[\sum_{i=1}^n a_i^2\right] = 0 \tag{7}$$

...and this is where I get stuck.

How can I solve (7) to show that:

$$a_i = x_i - \bar{x}$$

I realize that this is simply a question of algebraic manipulation (I think I've derived everything up to (7) correctly); I just can't see how to simplify it.

• At first, $\tilde \beta$ is not unbiased estimate because $E(\tilde \beta) = \alpha \sum a_i +\beta$. Nov 3 '18 at 2:45
• @a_statistician Thanks. I've added a missing detail to (2) which justifies the unbiasedness.
– Thev
Nov 3 '18 at 2:50
• eq. (5), should be partial derivative to $a_1, a_2,...,a_n$, so you have $n$ partial derivatives, and set them to 0, get the solution of$a_1,a_n$? I think so. Nov 3 '18 at 3:34
• We're dealing with well-behaved function, so the derivative of the sum is the sum of the derivatives. I don't understand your comment beyond that, I'm afraid.
– Thev
Nov 3 '18 at 3:53
• I think it should be: $\frac {\partial V}{\partial a_1} = 0$, ..., : $\frac {\partial V}{\partial a_n} = 0$ Nov 3 '18 at 5:18

Because $$\sum_{i=1}^n a_i = 0$$, $$\sum_{i=1}^n a_i x_i =\sum_{i=1}^n a_i x_i -\bar x \sum_{i=1}^n a_i = \sum_{i=1}^n a_i (x_i-\bar x)$$. So (4) can be written as $$V(\tilde{\beta}) = \frac{\sigma^2 \sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i x_i\right]^2}=\frac{\sigma^2 \sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i (x_i - \bar x)\right]^2}$$
According to Cauchy-Schwarz Inequality, we have $$\left[\sum_{i=1}^n a_i (x_i - \bar x)\right]^2 \le \sum_{i=1}^n (x_i-\bar x)^2\sum_{i=1}^n a_i^2$$ Divided by $$\left[\sum_{i=1}^n a_i (x_i - \bar x)\right]^2 \sum_{i=1}^n (x_i-\bar x)^2$$ on both sides, we get: $$\frac{\sum_{i=1}^n a_i^2}{\left[\sum_{i=1}^n a_i (x_i-\bar x)\right]^2} \ge \frac 1 {\sum_{i=1}^n (x_i-\bar x)^2} = \frac {\sum_{i=1}^n (x_i-\bar x)^2}{\left(\sum_{i=1}^n (x_i-\bar x)^2\right)^2}$$ So $$V(\hat \beta) \le V(\tilde \beta)$$
• very nice. another sneakier way is to argue that $\hat{\beta}$ is the MLE and MLE's minimize the residual variance. ( the sum of (yactual - predicted ). by their definition. i.e: maximum likelihood, in the case of the normal distribution, minimizes the variance of $\epsilon$ and the variance of $\beta$ is a function of the variance of $\epsilon$. I like your way better. Nov 3 '18 at 5:09