5
$\begingroup$

In several situations, I have two unbiased estimators, and I know one of them is better (lower variance) than the other. However, I would like to get as much information as possible, and I would like to do better than throwing out the weaker estimator.

$$\newcommand{\Outcome}{\text{Outcome}}\newcommand{\Skill}{\text{Skill}}\newcommand{\Luck}{\text{Luck}}\Outcome = \Skill + \Luck$$

$\Outcome$ is observed. $\Skill$ is what I would like to determine. $\Luck$ is known to have the average value $0$. From other observables, I can estimate $\Luck$ by $L$ so that $\mathbb E(L) = 0$ and $\mathrm{Var}(\Luck-L) \lt \mathrm{Var}(\Luck)$.

$\Outcome$ is an unbiased estimator for $\Skill$. A better estimate from variance reduction is $\Outcome - L$, which is also unbiased. For example, in one situation $\Outcome$ is the average of repeated trials, and I might produce a $95\%$ confidence interval of $[-5.0,13.0]$ without using variance reduction. Using variance reduction, I might get a confidence interval of $[-2.0,4.0]$.

The typical practice is for people to use $\Outcome-L$ instead of $\Outcome$. However, this is unsatisfactory to me because in my experience, there is more information in the pair $(\Outcome, \Outcome-L)$ than in just $\Outcome-L$. Specifically, in some situations I know that if $\Outcome$ is low, then $\Outcome-L$ tends to be an underestimate for $\Skill$, and if $\Outcome$ is high, then $\Outcome-L$ tends to be an overestimate for $\Skill$.

What's a good way to take advantage of the extra information from knowing both estimators?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
5
$\begingroup$

If you have two unbiased estimators $L_1$ and $L_2$ then accuracy is measured by variance (for a mean square error loss function).

Take $L=aL_1+(1-a)L_2$, a weighted average of the two with $0 < a < 1$.

$$\newcommand{\Var}{\mathrm{Var}}\newcommand{\Cov}{\mathrm{Cov}} \Var(L) =a^2\Var(L_1) +(1-a)^2\Var(L_2) +2a(1-a) \Cov(L_1, L_2) \>. $$
When $\Cov(L_1,L_2) \leq 0$, $$ \Var(L) \leq a^2 \Var(L_1) +(1-a)^2 \Var(L_2) < a \Var(L_1) +(1-a)\Var(L_2) \leq \max(\Var(L_1), \Var(L_2)) \>. $$

If $\Cov(L_1, L_2)$ is sufficiently negative then $\Var(L)$ will also be less than $\min(\Var(L_1), \Var(L_2))$.

If $a$ is not restricted to $[0,1]$ and $1-a$ is replaced by $b$ it is possible to find constants $a$ and $b$ such that $\Var(L) = \Var(a L_1) + \Var(b L_2)$ is less than $\min (\Var(L_1), \Var(L_2))$. For $\Cov(L_1, L_2)>0$ a particular choice of $a>0$ and $b<0$ will work. This shows that there is always a way to improve the accuracy of unbiased estimators if you can determine a linear combination of the two that lowers the variance. One will always exist. Clearly this is a good way to take advantage of the knowledge of both estimators.

Improve this answer
$\endgroup$
14
  • 3
    $\begingroup$ Questions to consider: (1) Why is "accuracy measured by variance" just because two estimators are unbiased? Why not any other convex symmetric loss function, for example? (Yes, it's convenient.) (2) Why are you only considering $a \in (0,1)$? In particular, what happens if they're negatively correlated and $L_1$ has much higher variance than $L_2$? (Hint: You can do better than $a \in (0,1)$.) (3) "Best" linear combinations of unbiased estimators are nice, but we should also remain aware that in some circumstances, there may be better nonlinear unbiased estimators. $\endgroup$
    – cardinal
    Commented Jul 17, 2012 at 4:21
  • 6
    $\begingroup$ Michael, I have tried to do a major cleanup of the math. Please check that I have not introduced errors. I would urge you to consider looking through some other answers of yours in which several other users have been kind enough to provide this service to you. Like any language, $\LaTeX$ can seem a little daunting at first, but it's not hard for the computer-adept such as yourself and the end product is orders of magnitude cleaner. :) $\endgroup$
    – cardinal
    Commented Jul 17, 2012 at 13:45
  • 2
    $\begingroup$ While I tend to drop hints rather than be explicit, maybe I can be more direct for a moment and point out that optimizing over $a$ for a linear combination of the first form you consider is a pretty routine calculation and yields an interesting result. Adding that explicit solution may also make the resulting answer more compact. :) $\endgroup$
    – cardinal
    Commented Jul 17, 2012 at 13:47
  • 2
    $\begingroup$ This answer is a tad obscure. It never states the bottom line in clear, English language: what is a good way to take advantage of the extra information from knowing both estimators? Also, how does it apply to the original poster's specific context? The reader is left to try to infer the answer. I think the answer might be improved by spelling it out.... $\endgroup$
    – D.W.
    Commented Jul 17, 2012 at 20:19
  • 2
    $\begingroup$ @D.W. I did further editing of my answer based on your critque. $\endgroup$
    – Michael R. Chernick
    Commented Jul 17, 2012 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.