# Finding an unbiased estimator with the smallest variance

I will state the question then my methodology. Q: We have 3 random variables, $$X1,X2,X3$$ that are independent and identically distributed (iid). We would like to estimate $$\theta = E[X1]$$. Suppose that we are considering an estimator of the following form: $$\hat{\theta}=a_1X1 + a_2X2 + a_3X3$$ where $$a=(a_1,a_2,a_3)$$ is a chosen set of numbers. Now find $$a1,a2,a3$$ such that $$\hat{\theta}$$ is unbiased and its variance is smallest among all choices of $$a$$.

My progress so far: We want the bias to be $$0$$ since we want this estimator to be unbiased so... $$E[a_1X1 + a_2X2 + a_3X3] - E(X1)=0$$ $$\mu(a_1+a_2+a_3-1)=0$$ so the sum of $$a1,a2,a3$$ should be $$1$$. This is the constraint.

Now we want the estimator with the lowest variance. So taking the variance of the estimator we have: $$Var{(\hat{\theta}}) = a_1^2\sigma^2+a_2^2\sigma^2+a_3^2\sigma^2$$

So I think we need to minimize the variance equation with the above constraint. The problem is I do not how to do that, is there another method or maybe I have done something wrong?

My econometric textbook did very little to explain the above.

• You need a method for constrained optimisation - Lagrange multipliers will solve this problem in about six lines of working. (A little more if you want to check the solution is actually a minimum for the variance). – Silverfish Jan 12 '15 at 5:18
• Right. If you're studying econometrics it is particularly important for you to see that "equally weighted" mean only minimized the variance of your estimator because the variances of the $X_i$ were equal. It is worth checking you can use Lagrange multipliers, or some other method, to minimize the variance in this situation, as this can easily extend to the unequal variance case where different weights are optimal. – Silverfish Jan 12 '15 at 5:30
• In particular, we "trust" an $X_i$ more if its variance is lower, because we expect its value to be closer to the true $\mu$. High variance $X_i$ contain less information about the true mean so we weight them less. But how much less, you can only really see if you can perform the calculation. In the present case, all are equal variance so they should get equal weights, and we can see that without extended calculation. – Silverfish Jan 12 '15 at 5:33
• @Silverfish It would be helpful to post an answer with the general method - I have the feeling this question may be a returning customer. – Alecos Papadopoulos Jan 12 '15 at 9:25
• (1) Are you equating "$\mu$" with "$\theta$"? (2) In general, this question is equivalent to minimizing the distance $\sqrt{a_1^2 + \cdots + a_n^2}$ between the point $(a_1,\ldots,a_n)$ and the origin, subject to $a_1+\cdots+a_n=1$: that is, to find the point on this affine plane closest to the origin. That is found in any number of standard ways. The simplest: geometry tells us the vector from the origin to the point must be perpendicular to the plane, instantly showing $a_1=a_2=\cdots=a_n$, etc. – whuber Jan 12 '15 at 16:34

We can justify the solution to this problem by symmetry. Being i.i.d., the observed $X_i$ are interchangeable, so their coefficients ought to be also. So it is no surprise they are equally weighted. Note that this symmetry argument only works in the case where your observations are of equal variance, and in econometrics you will study situations where this is not the case. It would be useful to learn some machinery that can handle the present case more formally, and can be adapted to handle such variations.

Problems in which you seek to optimize $f(a_1, a_2, \dots, a_n)$ subject to a constraint that $g(a_1, a_2, \dots, a_n) = c$ are well-suited to solution by Lagrange multipliers.

In particular you seek to minimize $f(a_1, a_2, a_3) = \sigma^2 a_1^2 + \sigma^2 a_2^2 + \sigma^2 a_3^2$ subject to $g(a_1, a_2, a_n) = a_1 + a_2 + a_3 = 1$ (so that $c=1$).

Defining $\Lambda = f + \lambda (g - c)$ we see that, since $g(a_1, a_2, a_3)-c = 0$, minimizing $f$ is equivalent to minimizing $\Lambda(a_1, a_2, a_3, \lambda)$. We find the partial derivatives and set them equal to zero:

$$\Lambda(a_1, a_2, a_3, \lambda) = \sigma^2 a_1^2 + \sigma^2 a_2^2 + \sigma^2 a_3^2 + \lambda (a_1 + a_2 + a_3 - 1)$$

$$\frac{\partial \Lambda}{\partial a_1} = 2 \sigma^2 a_1 + \lambda = 0$$ $$\frac{\partial \Lambda}{\partial a_2} = 2 \sigma^2 a_2 + \lambda = 0$$ $$\frac{\partial \Lambda}{\partial a_3} = 2 \sigma^2 a_3 + \lambda = 0$$ $$\frac{\partial \Lambda}{\partial \lambda} = a_1 + a_2 + a_3 - 1 = 0$$

From the partial derivatives with respect to each $a_i$ we can see $a_1 = a_2 = a_3 = -\frac{\lambda}{2 \sigma^2}$, and since each must be equal, the final partial derivative (which is equivalent to the original constraint) ensures that $a_1 = a_2 = a_3 = \frac{1}{3}$. So we must weight the observations $x_1$, $x_2$ and $x_3$ equally when taking their mean.

A little bit of thought shows this generalizes to $n$ observations; for $i=1, 2, \dots, n$ the partial derivative with respect to $a_i$ is simply:

$$\frac{\partial \Lambda}{\partial a_i} = 2 \sigma^2 a_i + \lambda = 0$$

Hence $a_i = -\frac{\lambda}{2 \sigma^2}$ for each $i$, so the weights are equal. Moreover, the final partial derivative would be:

$$\frac{\partial \Lambda}{\partial \lambda} =\sum_{i=1}^{n} a_i - 1 = 0$$

Since the weights total to one, each must be $a_i = \frac{1}{n}$.

Suppose instead that the variances of your observations were unequal, $\text{Var}(X_i)=\sigma^2_i$. Now we minimize $f(a_1, a_2, a_3) = \sigma_1^2 a_1^2 + \sigma_2^2 a_2^2 + \sigma_3^2 a_3^2$ subject to $g(a_1, a_2, a_n) = a_1 + a_2 + a_3 = 1$. This time,

$$\frac{\partial \Lambda}{\partial a_i} = 2 \sigma_i^2 a_i + \lambda = 0$$ $$\frac{\partial \Lambda}{\partial \lambda} =\sum_{i=1}^{n} a_i - 1 = 0$$

Since each $a_i = -\frac{\lambda}{2 \sigma_i^2}$ we can see $a_i \propto \frac{1}{\sigma^2_i}$. So observations with a higher variance are given a lower weighting - this makes sense, as they are in some sense less trustworthy. We give larger weightings to observations with lower variances, since they convey more information about the true value of $\mu$. To ensure weights sum to one, the formula for the weights is:

$$a_i = \frac{1}{\sigma^2_i} \left( \sum_{j=1}^n \frac{1}{\sigma^2_j} \right)^{-1}$$

A confession: at this point I have simply claimed that the weights are "optimal", without checking whether I minimized or maximized the variance! Actually, it is possible to see that the solutions obtained so far are minima in various ways, for example geometrically. But since you are studying econometrics, the method I expect you will encounter is the bordered Hessian, which in the general case, with one constraint, appears as:

$$H^B(f,g) = \begin{vmatrix} 0 & \dfrac{\partial g}{\partial a_1} & \dfrac{\partial g}{\partial a_2} & \cdots & \dfrac{\partial g}{\partial a_n} \\[2.2ex] \dfrac{\partial g}{\partial a_1} & \dfrac{\partial^2 f}{\partial a_1^2} & \dfrac{\partial^2 f}{\partial a_1\,\partial a_2} & \cdots & \dfrac{\partial^2 f}{\partial a_1\,\partial a_n} \\[2.2ex] \dfrac{\partial g}{\partial a_2} & \dfrac{\partial^2 f}{\partial a_2\,\partial a_1} & \dfrac{\partial^2 f}{\partial a_2^2} & \cdots & \dfrac{\partial^2 f}{\partial a_2\,\partial a_n} \\[2.2ex] \vdots & \vdots & \vdots & \ddots & \vdots \\[2.2ex] \dfrac{\partial g}{\partial a_n} & \dfrac{\partial^2 f}{\partial a_n\,\partial a_1} & \dfrac{\partial^2 f}{\partial a_n\,\partial a_2} & \cdots & \dfrac{\partial^2 f}{\partial a_n^2} \end{vmatrix}$$

Note that I am following Binmore and Davies, Calculus: Concepts and Methods, in calling the determinant the Hessian, rather than the matrix itself (as Wikipedia does). Observe that the upper left 1-by-1 submatrix of our ($n+1$)-by-($n+1$) matrix is just zero - corresponding to $\frac{\partial^2 \Lambda}{\partial \lambda^2} = 0$. If in even fuller generality there had been $m$ constraints to apply, then the upper left $m$-by-$m$ submatrix of our ($n+m$)-by-($n+m$) matrix would contain only zeroes. The signs of the leading principal minors (determinants of upper left submatrices) are often sufficient determine whether the solution is a minimum or maximum. The only informative ones consist of the first $2m+1$ rows and columns, or more. If they all have the same sign, and this sign is that of $(-1)^m$, then we have a minimum. If they alternate in signs, with the smallest having sign $(-1)^{m+1}$ (or alternatively, the largest - i.e. the whole bordered Hessian - has sign $(-1)^n$) then we have a maximum. This is covered in Wikipedia but for more detail see e.g. these lecture notes (or many others which can be found by an internet search).

I will return our attention to your case where $m=1$ and $n=3$, and the bordered Hessian is:

$$H^B(f,g) = \begin{vmatrix} 0 & 1 & 1 & 1 \\[2.2ex] 1 & 2\sigma^2 & 0 & 0 \\[2.2ex] 1 & 0 & 2\sigma^2 & 0 \\[2.2ex] 1 & 0 & 0 & 2\sigma^2 \end{vmatrix}$$

In this case $2m+1=3$ so we only need to consider the principal leading minors of sizes 3-by-3 and 4-by-4. The latter is just $H^B_4 = H^B$ itself, and the former is:

$$H^B_3 = \begin{vmatrix} 0 & 1 & 1 \\[2.2ex] 1 & 2\sigma^2 & 0 \\[2.2ex] 1 & 0 & 2\sigma^2 \end{vmatrix}$$

This determinant can easily be reduced to that for a lower-triangular matrix by subtracting $\frac{1}{2 \sigma^2}$ times row 2 from row 1, then subtracting the same multiple of row 3 from row 1:

$$H^B_3 = \begin{vmatrix} -\frac{1}{\sigma^2} & 0 & 0 \\[2.2ex] 1 & 2\sigma^2 & 0 \\[2.2ex] 1 & 0 & 2\sigma^2 \end{vmatrix} = -4 \sigma^2 < 0$$

By the same procedure, we can show the deteminant $H^B_4$ is negative. The relevant leading principal minors had the same sign as each other and also of $(-1)^m = (-1)^1 = -1$ and so we can see our solution is indeed the minimum variance. There is no difficulty applying the same lower-triangularization technique when there are $n$ observations instead of three, or even to the case with unequal variances: all the appropriate minors are clearly negative after triangularization, since the diagonal entries are positive with the exception of the top-left entry, which will be negative as it is zero subtract various positive multiples of one.

• @nicefella I added something about the case with unequal variances. The apparatus I present in this question is in some sense too "heavy-duty" - there are nicer ways of seeing the result (as I hope mine and whuber's comments on the original question showed). However, tasks like yours are often assigned in introductory econometrics classes where the Lagrange multiplier is likely to be the tool of choice. – Silverfish Jan 12 '15 at 21:43
• (+1) I would only write the constraint in the Lagrangean as $(c-g)$ and change subsequent signs, so that in the first-order conditions a positive value for the multiplier is implied. Although not necessary in the case of only equality constraints, it aligns better with the next step (inequality constraints). Your work with the bordered Hessian is detailed, and I didn't spot any mistakes. Of course you could also mention the shortcuts:if the objective function is convex and the constraints linear, then the necessary conditions are also sufficient, etc. – Alecos Papadopoulos Jan 12 '15 at 21:58
• @alecos I originally wrote $(c-g)$ but back-pedalled when I realised the Wikipedia articles I was referring the OP to consistently had it the other way around. Incidentally: do you end up with negative derivatives of $g$ in your bordered Hessian? I've seen some people use the ($c-g$) notation but then switch to having positive derivatives of $g$ in $H^B$. (Since this is equivalent to multiplying both a row and a column by -1, clearly this makes no difference to any of the relevant determinants, but I find inconsistent notation a pain to follow!) – Silverfish Jan 12 '15 at 22:11
• Familiarization with the Lagrange multipliers is more important than the mechanics of the bordered Hessian. With the constraint written as $c-g$, then the condition for a minimum is that all principal minors of the bordered Hessian are negative, implying that the Hessian itself is positive definite. For the first leading principal minor, this indeed requires that the derivatives of $g$ are positive. – Alecos Papadopoulos Jan 12 '15 at 22:20
• @alecos Doesn't someone who writes $\begin{vmatrix} 0 & g_x & g_y \\ g_x & L_{xx} & L_{xy} \\ g_y & L_{yx} & L_{yy} \end{vmatrix}$ follow the same reasoning (i.e. looks for the same patterns in the signs of the principal leading minors) as someone who writes $\begin{vmatrix} 0 & -g_x & -g_y \\ -g_x & L_{xx} & L_{xy} \\ -g_y & L_{yx} & L_{yy} \end{vmatrix}$? – Silverfish Jan 12 '15 at 22:31