10
$\begingroup$

Suppose I have i.i.d. samples $X_1, \cdots, X_n$ from some unknown distribution $F$ and I wish to estimate the mean $\mu=\mu(F)$ of that distribution and I insist that the estimator be unbiased - i.e., $\mathbb{E}[T(X_1, \cdots, X_n)] = \mu$.

The canonical estimator is the sample mean $\overline{X} = \frac{1}{n} \sum_{i=1}^n X_i$. This is always unbiased and for many families of distributions, such as Gaussians, it is optimal or near-optimal in terms of variance.

However, the sample mean is not robust. In particular, the sample mean can change arbitrarily if a single $X_i$ is changed. This means it has a breakdown point of 0.

A more robust estimator is the sample median. Changing a few data points will not, for most samples, significantly change the median. This has a breakdown point of 0.5, which is the highest possible.

For Gaussian data, the sample median has higher variance than the sample mean (by a factor of $\pi/2$). However, for other distributions, such as the Laplace distribution or Student's $t$-distribution, the median actually has lower variance than the mean.

Furthermore, the median is always unbiased if the distribution is symmetric (about its mean). Many natural distributions are symmetric, but many are not, such as the following examples.

  1. Binomial
  2. Poisson
  3. log-Normal
  4. Gamma
  5. F-distribution
  6. Geometric distribution

My question is: Are there robust and unbiased estimators for the means of natural asymmetric distributions? By robust I simply mean a non-zero breakdown point and by natural I mean something from the above list or similar (just not a concocted example). I can't find any examples. I would be particularly interested in the Binomial case.

$\endgroup$
2
  • 4
    $\begingroup$ (+1) Because robust estimation is primarily of interest in nonparametric settings, likely little has been done along these lines. However, under your parametric models it's straightforward to concoct robust unbiased estimators out of order statistics: that will give you plenty of examples to consider. $\endgroup$
    – whuber
    Mar 23 '19 at 18:16
  • 1
    $\begingroup$ There is probably an asymmetric trimmed mean that would work in every one of those cases. However, I suspect the (asymmetric) trimming proportions would need to be derived on a case-by-case basis (and no, I've never seen it done). Of incidental interest, I asked a question about this in the non-parametric case, but there probably isn't an answer in that scenario. $\endgroup$ May 30 '19 at 5:04
2
$\begingroup$

As already said by whuber, one way to answer your question is to de-biase your estimator. If the robust estimator is biased, maybe you can subtract the theoretical bias (according to a parametric model), there are some work that try to do that or to subtract an approximation of the bias (I don't remember a ref but I could search for it if you are interested). For instance, think about the empirical median in an exponential model. We can compute its expectation and then substract this expectation, if you want I can make the computations this is rather simple ... this becomes more difficult if the estimator is more complicated than the median and this works only in parametric models.

A maybe less ambitious question is whether we can construct a consistent robust estimator. This we can do but we have to be careful of what we call robust.

If your definition of robust is having a non-zero asymptotic breakdown point, then already we can prove that this is impossible. Suppose that your estimator is called $T_n$ and it converges to $\mathbb{E}[X]$. $T_n$ has a non-zero breakdown point which means that there can be a portion $\varepsilon>0$ of the data arbitrarily bad and nonetheless $T_n$ will not be arbitrarily large. But this can't be because at the limit, if a portion of the data is an outlier, this translates: with probability $1-\varepsilon$, $X$ is sampled from the target distribution $P$ and with probability $\varepsilon$ $X$ is arbitrary, but this makes $\mathbb{E}[X]$ arbitrary also (if you want me to put it formally, I can) which is in contradiction with the non-asymptotic breakdown point of $T_n$.

Finally, to conclude on this, we can take the non-asymptotic point of view. Saying that we don't care about the asymptotic breakdown point, what is important is either a the non-asymptotic breakdown point (something like a breakdown point of $1/\sqrt{n}$. Or to be efficient on heavy-tailed data.

In this case, there are estimators that are robust and consistent estimators of $\mathbb{E}[X]$. For instance, we can use Huber's estimator with a parameter that goes to infinity or we can use the median-of-means estimator with a number of blocks that tends to infinity. References for this line of thought are "Challenging the empirical mean and empirical variance: A deviation study" by Olivier Catoni or "Sub-Gaussian mean estimators" by Devroye et al (these ref are in the theoretical community, they may be complicated if you are not familiar with empirical processes and concentration inequalities).

$\endgroup$
3
  • 1
    $\begingroup$ I think you may be demanding too much of the estimator. When you mix in an arbitrary contaminating distribution to assess robustness, the concept of "unbiased" really doesn't apply any more, because you have changed the distributional assumption. It is possible for an estimator to be unbiased for a specified parametric family while still being robust relative to an arbitrary contaminating distribution. $\endgroup$
    – whuber
    Sep 20 '20 at 15:13
  • $\begingroup$ It is exactly what I said (or what I beleived I said), I said it was impossible even to be consistent with these hypothesis. Please show me where I am not clear enough in my explications. $\endgroup$
    – TMat
    Sep 21 '20 at 6:07
  • $\begingroup$ You are right, it was not so clear, the breakdown point must tend to 0 and hence the corruption must tend to zero for our estimator to work. $\endgroup$
    – TMat
    Sep 21 '20 at 14:05
1
$\begingroup$

This is not an unbiased estimate, but it is consistent (you can let the bias approach to zero as the sample size grows).

You can take a trimmed sample (remove the highest and lowest values) and use the mean of the trimmed sample as the estimate.

In the case of a know distribution then you might use an appropriate scaling to make the estimate less biased (or not biased at all), or otherwise the bias will just decrease when you take smaller samples.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.