# How to show the least square estimator of $b$ has the minimum variance in the class $\sum a_iy_i$

Consider the regression model: $$y_i=bx_i+e_i,1\leq i\leq n.$$

where $x_i$'s are fixed non-zero real numbers and $e_i$'s are independent random variables with mean zero and equal variance.

$(a)$Consider estimator of the form $T=\sum_{i=1}^{n}a_iy_i$(where $a_i$'s are non random real numbers)that are unbiased for $b$.Show that the least square estimator of $b$ has the minimum variance in this class of estimators.

Minimizing $\sum e_i^{2}=\sum(y_i-bx_i)^2$ $w.r.t$ $b$,we get the least square estimate of $b$ as, $$\hat b=\frac{\sum y_ix_i}{\sum x_i^{2}}$$ Now,$var(\hat b)=\frac{\sigma^{2}}{\sum x_i^{2}}$,where $\sigma^2=var(y),$I assume.

Now,I tried to minimize $Var(T)=\sum a_i^2\sigma^{2}$ subject to $\sum a_i=1$ and I am getting $a_i=\frac{1}{n}$,i.e.,$T=\bar y$ as the $BLUE$ of $b$.I don't know where I am making the mistake.May be,my approach is not right.

• $T=\hat b$ in your problem, not $T=\bar y$. – Christoph Hanck Jun 6 '16 at 14:22
• @ChristophHanck: We know that least square estimate is the BLUE.Could you please tell me how can we find the BLUE from the class $\sum a_iy_i$ and show that it is equal to $\hat b?$ – priyanka Jun 6 '16 at 14:35
• Have you heard of the Gauß-Markov theorem? – Christoph Hanck Jun 6 '16 at 15:13

## 1 Answer

Your minimum variance problem is: $$\begin{equation} \begin{array}{*2{>{\displaystyle}r}} \mbox{minimize (over a_i)} & \mathrm{Var}\left( \sum_i a_i y_i \; \middle| \; \{x_i\}\right) \\ \mbox{subject to} & \mathrm{E}\left[\sum_i a_iy_i \; \middle|\;\{x_i\}\right] = b \end{array} \end{equation}$$ Remember you can always substitute $y_i = b x_i + \epsilon_i$ $$\begin{equation} \begin{array}{*2{>{\displaystyle}r}} \mbox{minimize (over a_i)} & \mathrm{Var}\left( \sum_i a_i (bx_i + \epsilon_i) \; \middle| \; \{x_i\}\right) \\ \mbox{subject to} & \mathrm{E}\left[\sum_i a_i(bx_i + \epsilon_i) \; \middle|\;\{x_i\}\right] = b \end{array} \end{equation}$$

Simplifying, you can show this is an equivalent problem to: $$\begin{equation} \begin{array}{*2{>{\displaystyle}r}} \mbox{minimize (over a_i)} & \sum_i a_i^2 \\ \mbox{subject to} & \sum_i a_i x_i = 1 \end{array} \end{equation}$$

Which will have a nice solution! You'll see that the solution $\mathbf{a}^*$ to the above optimization problem will make your estimator $T$ equal to $\hat{b}$.