4
$\begingroup$

Problem:

Assume that we want to estimate $f(\theta)$ with a pre-specified strictly increasing function $f$ and a parameter $\theta$.

Let $\hat{\theta}_1$ and $\hat{\theta}_2$ be unbiased estimators for $\theta$. My question is to compare which estimator is better for $f(\theta)$:

  1. $f\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right)$, or
  2. $\frac{f(\hat{\theta}_1) + f(\hat{\theta}_2)}{2}$.

My approach:

My approach is taking an expectation on both based on Taylor's expansion of $f$, provided that they are well-defined.

  1. $$\mathbb{E}f\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) = f(\theta) + \frac{f''(\theta)}{2}\text{Var}\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) + \ldots $$

  2. $$\mathbb{E} \left[\frac{f(\hat{\theta}_1) + f(\hat{\theta}_2)}{2} \right] = f(\theta) + \frac{f''(\theta)}{2}\left(\frac{\text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2)}{2}\right) + \ldots $$

Here, the first-order disappears in each expectation because $\hat{\theta}_1$ and $\hat{\theta}_2$ are unbiased.

Let's further assume $f''(\theta)>0$. Then, since $$\label{varineq} (0 \le) \text{Var}\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) \le \frac{\text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2)}{2}, $$ the first estimator (i.e. a function of averages) seems less distant from $f(\theta)$, as long as the remaining terms (order $\ge 3$) are ignored. The last inequality comes from a simple observation: $$ \frac{\text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2)}{2} - \text{Var}\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) = \dfrac{1}{4}\left( \text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2) - 2\text{Cov}\left(\hat{\theta}_1, \hat{\theta}_2\right) \right) = \dfrac{\text{Var}(\hat{\theta}_1 - \hat{\theta}_2)}{4} \ge 0. $$

Question:

I wonder if this is logical and there is another way to justify a better estimator.

Edit:

Let's restrict the class of $f$ by the strictly increasing ones.

$\endgroup$
  • 1
    $\begingroup$ An example would help, e.g.: Suppose someone gives you the mean $\theta_1$, median $\theta_2$, and size $n$ of a sample drawn from a normal parent $N(\mu,\sigma)$. What are your best estimates for $\mu$ and $\mu^2$? It should be possible to work this out using the joint distribution at math.stackexchange.com/questions/477115/… -- note the answers may well be weighted averages and the covariance may be significant. $\endgroup$ – Matt F. Mar 2 '20 at 14:51
  • 2
    $\begingroup$ Where do you obtain the last inequality about variances? It's not generally true. Counterexamples include the cases $\hat\theta_2 = -\hat\theta_1$ (for which the right hand side is zero) and when the $\hat\theta_i$ are independent (for which the right hand side is half the left hand side). $\endgroup$ – whuber Mar 2 '20 at 14:54
  • $\begingroup$ @whuber Oops, sorry for that. I used the opposite inequality. Now it is edited. $\endgroup$ – inmybrain Mar 2 '20 at 19:43
  • 2
    $\begingroup$ Bear in mind that if the function represents something other than a linear transformation of the inputs, the two will yield [different] answers to different questions. To get a sense of that, it may help to read my answer here: Difference between generalized linear models & generalized linear mixed models. $\endgroup$ – gung - Reinstate Monica Mar 2 '20 at 19:57
  • $\begingroup$ @gung-ReinstateMonica Thanks! Then, let's keep our focus on the strictly increasing $f$ (see my edit). Does this make sense to you? $\endgroup$ – inmybrain Mar 2 '20 at 20:19
0
$\begingroup$

Suppose $\theta_1$ and $\theta_2$ are unbiased estimators of $\mu$ with a bivariate normal distribution.

Let $\sigma_1 = k \sigma_2$, with $k<1$, so $\theta_1$ is the more efficient estimator.

Let the correlation be $r$, with $-1<r<1$.

We seek the least-variance linear estimates for $\mu$ and $\mu^2$. The results are:

\begin{align} \min_{a,b}Var[\mu-(a\theta_1+b\theta_2)] \text{ has }&\mu \sim \frac{(1-kr)\theta_1+(k^2-kr)\theta_2}{1-2kr+k^2}\\ \min_{c,d,e}Var[\mu^2-(c\theta_1^2+d\theta_1\theta_2+e\theta_2^2)] \text { has }&\mu^2 \sim \left(\frac{(1-kr)\theta_1+(k^2-kr)\theta_2}{1-2kr+k^2}\right)^{\!2} \end{align}

So for functions $f$ that are quadratic or well approximated by quadratics, if $\frac{\theta_1+\theta_2}{2}$ is the best linear estimate for $\mu$, then $f\left(\frac{\theta_1+\theta_2}{2}\right)$ is the best linear estimate for $f(\mu)$. In general an appropriate weighted average inside $f$ works even better.

Example: $\theta_1$ is the mean of a sample from a normal parent with mean $\mu$, and $\theta_2$ is the median from the same sample. Then $k=r=\sqrt{2/\pi}$, and the above formula shows that the best estimate for $\mu$ is $\theta_1$, and the best estimate for $\mu^2$ is $\theta_1^2$, i.e. ignoring the median.

Example: $\theta_1$ is the mean of a sample from a normal parent with mean $\mu$, and $\theta_2$ is the median of a different sample of the same size. Then $k=\sqrt{2/\pi}$, $r=0$, so the best estimate for $\mu$ is $(\pi\theta_1+2\theta_2)/(\pi+2)$, or $.611\theta_1 + .389\theta_2$, and the best estimate for $\mu^2$ is the square of that.

Example: $\theta_1$ is the mean of a sample from a normal parent with mean $\mu$, and $\theta_2$ is the mean of a different sample of the same size. Then $k=1$, $r=0$, so the best estimate for $\mu$ is $(\theta_1+\theta_2)/2$, and the best estimate for $\mu^2$ is $\left(\frac{\theta_1+\theta_2}2\right)^{\!2}$.

$\endgroup$
  • $\begingroup$ Thanks for your answer! I was more curious about general conditions of $f$ or the estimators, but this also helps. $\endgroup$ – inmybrain Mar 4 '20 at 14:10
  • $\begingroup$ What functions $f$ do you have in mind that aren't well approximated by quadratics, or what estimators do you have in mind that aren't distributed normally? $\endgroup$ – Matt F. Mar 4 '20 at 14:59
  • $\begingroup$ @inmybrain, if this is worthy of thanks, it is also worth an upvote, or a clearer specification of what situations you care about that it doesn't cover? $\endgroup$ – Matt F. Mar 5 '20 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.