9
$\begingroup$

Suppose that I have $n$ positive parameters to estimate $\mu_1,\mu_2,...,\mu_n$ and their corresponding $n$ unbiased estimates produced by the estimators $\hat{\mu_1},\hat{\mu_2},...,\hat{\mu_n}$, i.e. $\mathrm E[\hat{\mu_1}]=\mu_1$, $\mathrm E[\hat{\mu_2}]=\mu_2$ and so on.

I would like to estimate $\mathrm{min}(\mu_1,\mu_2,...,\mu_n)$ using the estimates at hand. Clearly the naive estimator $\mathrm{min}(\hat{\mu_1},\hat{\mu_2},...,\hat{\mu_n})$ is biased lower as $$\mathrm E[\mathrm{min}(\hat{\mu_1},\hat{\mu_2},...,\hat{\mu_n})]\leq \mathrm{min}(\mu_1,\mu_2,...,\mu_n)$$

Suppose that I also have the covariance matrix of the corresponding estimators $\mathrm{Cov}(\hat{\mu_1},\hat{\mu_2},...,\hat{\mu_n}) = \Sigma$ at hand. Is it possible to get an unbiased (or a less biased) estimate of minimum using the given estimates and the covariance matrix?

$\endgroup$
  • $\begingroup$ Are you willing to use Bayesian MCMC approach or you need some closed-form formula? $\endgroup$ – Martin Modrák Jul 9 '18 at 13:13
  • $\begingroup$ But a simple sampling approach is OK? (also, you don't strictly need priors for Bayesian analysis, but that's another story) $\endgroup$ – Martin Modrák Jul 9 '18 at 13:49
  • $\begingroup$ @MartinModrák I am not experienced with sampling approaches. If I do bayesian I usually do simple conjugate stuff. But if you think this is the way to go I will go ahead and learn. $\endgroup$ – Cagdas Ozgenc Jul 9 '18 at 13:57
  • $\begingroup$ What else do you know about these estimates? Do you know the expressions? Do you know the distribution of the data used to estimate these parameters? $\endgroup$ – wij Jul 9 '18 at 14:36
  • $\begingroup$ @wij I can try to estimate some other moments of the estimators if needed. I don't have an analytical expression for the distribution of the estimators. Solution should not be (as a requirement of mine) dependent on the distribution of the data itself. $\endgroup$ – Cagdas Ozgenc Jul 9 '18 at 14:38
4
+75
$\begingroup$

I have no clear answer about existence of unbiased estimator. However, in terms of estimation error, estimating $\min(\mu_1, \dots, \mu_n)$ is a intrinsically difficult problem in general.

For instance, let $Y_1, \dots, Y_N \sim N(\mu, \sigma^2I)$, and $\mu = (\mu_1, \dots, \mu_n)$. Let $\theta = \min_i \mu_i$ be the target quantity and $\hat{\theta}$ is an estimate of $\theta$. If we use the "naive" estimator $\hat{\theta} = \min_i(\bar{Y}_i)$ where $\bar{Y_i} = \frac{1}{N}\sum_{j=1}^N Y_{i,j}$, then, the $L_2$ estimate error is upper bounded by $$ \mathbb{E}[\hat{\theta} - \theta]^2 \lessapprox \frac{\sigma^2\log n}{N} $$ up to constant. (Note that the estimate error for each $\mu_i$ is $\frac{\sigma^2}{N}$). Of course, if $\mu_i$'s are far away from each others and $\sigma$ is very small, the estimate error should be reduced to $\frac{\sigma^2}{N}$. However, in worst case, there is no estimate of $\theta$ works better than the naive estimator. You can precisely show that $$ \inf_{\hat{\theta}} \sup_{\mu_1, \dots,\mu_n} \mathbb{E}[\hat{\theta} - \theta]^2 \gtrapprox \frac{\sigma^2\log n}{N} $$ where the infimum takes over all possible estiamte of $\theta$ based on the sample $Y_1,\dots, Y_N$ and the supremum takes over all possible configuration of $\mu_i$'s.

Therefore the naive estimator is minimax optimal up to constant, and there is no better estimate of $\theta$ in this sense.

$\endgroup$
  • $\begingroup$ The additional information supplied is not helping at all? Which additional statistics can be helpful? $\endgroup$ – Cagdas Ozgenc Jul 11 '18 at 20:43
  • $\begingroup$ Sorry for making a confusing point. I did not mean the additional information (covariance) is not helpful. I just wanted to point out estimating minimum of several population means is difficult in nature. Covariance information should be helpful. For instance, in Normal case, if we have perfect correlations for all possible pairs, it means the random observations come from different mean + a common noise term. In this case, the naive estimator (minimum of sample means) is unbiased. $\endgroup$ – JaeHyeok Shin Jul 12 '18 at 1:06
3
$\begingroup$

EDIT: The following answers a different question than what was asked - it is framed as if $\mu$ is considered random, but does not work when $\mu$ is considered fixed, which is probably what the OP had in mind. If $\mu$ is fixed, I don't have a better answer than $\min(\hat\mu_1,...,\hat\mu_n)$


If we only consider estimates for mean and covariance, we can treat $(\mu_1, ..., \mu_n)$ as a single sample from multivariate normal distribution. A simple way to get an estimate of the minimum is then to draw a large number of samples from $MVN(\hat{\mu}, \Sigma)$, calculate the minimum of each sample and then take the mean of those minima.

The above procedure and its limitations can be understood in Bayesian terms - taking the notation from Wikipedia on MVN, if $\Sigma$ is the known covariance of the estimators and we have one observation, the joint posterior distribution is $\mu \sim MVN(\frac{\hat{\mu} + m \lambda_0}{1 + m}, \frac{1}{n+m} \Sigma)$ where $\lambda_0$ and $m$ arise from the prior where, before observing any data we take the prior $\mu \sim MVN(\lambda_0, m^{-1} \Sigma$). Since you are probably not willing to put priors on $\mu$, we can take the limit as $m \rightarrow 0$, resulting in flat prior and the posterior becomes $\mu \sim MVN(\hat{\mu}, \Sigma)$. However, given the flat prior we are implicitly making the assumption that the elements of $\mu$ differ a lot (if all real numbers are equally likely, getting similar values is very unlikely).

A quick simulation shows that the estimate with this procedure slightly overestimates $min(\mu)$ when the elements of $\mu$ differ a lot and underestimates $min(\mu)$ when the elements are similar. One could argue that without any prior knowledge this is correct behavior. If you are willing to state at least some prior information (e.g. $m = 0.1$), the results could become a bit better behaved for your use case.

If you are willing to assume more structure, you might be able to choose a better distribution than multivariete normal. Also it might make sense to use Stan or other MCMC sampler to fit the estimates of $\mu$ in the first place. This will get you a set of samples of $(\mu_1, ..., \mu_n)$ that reflect the uncertainty in the estimators themselves, including their covariance structure (possibly richer than what MVN can provide). Once again you can than compute the minimum for each sample to get a posterior distribution over minima, and take the mean of this distribution if you need a point estimate.

$\endgroup$
  • $\begingroup$ Note that I am not trying to estimate the minimum of N random variables. I am trying to estimate the minimum of N parameters. It seems your suggestion is an estimation for $E[min(\hat{\mu_1},\hat{\mu_2},...,\hat{\mu_n})]$ whereas I need an estimate for $min(\mu_1,\mu_2,...,\mu_n)$ $\endgroup$ – Cagdas Ozgenc Jul 9 '18 at 14:32
  • $\begingroup$ I tried to edit the answer to explain the rationale, hope that helps. $\endgroup$ – Martin Modrák Jul 9 '18 at 18:50
  • $\begingroup$ So is this sampling method yielding better results compare to the simple $min(\hat{\mu_1},\hat{\mu_2},...,\hat{\mu_n})$ estimator, which also works well when $\mu_i$ are far apart and underestimates when they are close. For it to be useful it should work when they are close. $\endgroup$ – Cagdas Ozgenc Jul 9 '18 at 19:02
  • $\begingroup$ Also note that all $\mu_i$ are positive numbers, so you don't really need the negative part of the real line. $\endgroup$ – Cagdas Ozgenc Jul 9 '18 at 19:43
  • 1
    $\begingroup$ You are correct that I ignore the signs and I don't see a simple way to accomodate them. Also the estimator I proposed does perform better when $\mu$ is considered random, but it is worse than $min(\hat{\mu})$ for fixed $\mu$. I don't think I can salvage this and I am unsure what is the best way forward - I am inclined to try to delete the answer as it does not really answer the question, but (I hope) the answer also contains some ideas that might be useful to somebody. $\endgroup$ – Martin Modrák Jul 11 '18 at 6:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.