# Estimator that is optimal under all sensible loss (evaluation) functions

Consider a probability distribution $$D$$ with a parameter $$\theta$$ and an i.i.d. sample $$S$$ from that distribution. I am interested in an estimator $$\hat\theta(S)$$ of $$\theta$$ that satisfies the following condition: $$\hat\theta(S) = \arg \min_{\hat\theta(S) \in \Theta(S)} \mathbb{E}\left( L(\hat\theta-\theta) \right)$$ for all loss functions $$L$$ such that $$L$$ is monotonically nonincreasing in $$(-\infty,0)$$, has a value of zero at 0, is monotonically nondecreasing in $$(0,+\infty)$$, and has a positive value somewhere away from zero*, where $$\Theta(S)$$ is a set of all possible estimators based on the sample $$S$$ and the expectation is taken over the possible i.i.d. samples.

Does there exist a special term for such an estimator in the statistics literature? Where could I learn more about such an estimator (conditions for its existence, its properties, and some examples)?

For instance, originally I guessed that $$\hat\theta$$ defined as the empirical mean of $$S$$ would be such an estimator when $$\theta$$ is the expectation and $$D$$ is normal. In other words, I guessed that the maximum likelihood estimator of the location parameter of the Gaussian distribution would satisfy the condition. But it seems I am wrong because for this estimator the condition cannot hold uniformly over all possible values of $$\Theta(S)$$, as pointed out by @CagdasOzgenc (as of now @CowboyTrader).

My question is motivated, among other, by statements such as this:

For example, in cases with a linear forecasting model and data that are jointly normal in the outcome and predictor variables, under MSE loss the forecasting model minimizes the equivalent of the negative of the maximum likelihood estimator (MLE). Assuming that the model is known and the variables are joint normally distributed, the MLE is an efﬁcient estimator of the model parameters, regardless of the loss function. One can then proceed by simply plugging the maximum likelihood parameter estimates into the optimal forecast for the problem that still involves the correct loss function.

(Elliott and Timmermann, 2016) (emphasis is mine).

*$$L$$ could be strictly decreasing to the left of 0 and strictly increasing to the right of 0 if that makes it easier, but I would prefer a more general $$L$$ as above.

References:

• Do you want to do something about the trivial loss function $L=0$? Nov 2 '17 at 9:38
• I don't think such a "universally optimal" estimator that is independent of $L$ exists, except for some corner cases, like estimating off $n=1$ datasets. Take a family of loss functions $L_\tau$ parameterized by $\tau\in[0,1]$, which are minimized by the $\tau$th quantile of the raw data (e.g., en.wikipedia.org/wiki/Quantile_regression). Then your optimal estimator will be this $\tau$th quantile, and in particular depend on $L_\tau$ (more specifically, on $\tau$). Maybe I am missing something? Nov 2 '17 at 9:53
• @CagdasOzgenc, definitely, it stems from there. I welcome all answers and cannot discourage any before I see them. Nov 2 '17 at 10:17
• I don't see how such an estimator could possibly exist: if it did, that would demonstrate that the loss function doesn't matter. But of course it does: by making the loss increase more rapidly for positive losses than negative ones, for instance, you will favor underestimating $\theta$, leading to a different optimum. Am I misunderstanding what you're asking? I see I am repeating @StephanKolassa's objection--I think it's a solid one. This has nothing to do with prediction and everything to do with estimation.
– whuber
Nov 2 '17 at 14:05
• @RichardHardy: you might stand a better chance of having your paper published if you could establish that such an estimator does indeed exist for at least some class of loss function. (There is the old story about a mathematician who proved a whole paper of wonderful properties about some structure - but where a reviewer pointed out that there were no non-trivial examples of the structure in question.) Nov 2 '17 at 18:11

### Universally Uniformly Best Unbiased Estimator

If you consider unbiased estimators and convex loss functions then you can consider the universally uniformly best unbiased estimator (UUBUE).

From "Pinelis, Iosif. A characterization of best unbiased estimators. arXiv preprint arXiv:1508.07636 (2015)."

A statistic $$T$$ is called universally uniformly best unbiased estimator (UUBUE) if it is $$\mathcal{L}$$-UBUE for all convex loss functions $$\mathcal{L}$$.

...

Proposition 9. Take any statistic $$T$$ and any loss function $$\mathcal{L} \in \mathscr{C}$$ . Then $$T$$ is a UMVUE iff $$T$$ is an L-UBUE iff $$T$$ is UUBUE.

The proof of this proposition is ascribed to L.B. Klebanov (Unbiased estimates and convex loss functions translated in 1978) and L. Schmetterer and H. Strasser (Zur Theorie der erwartungstreuen Schätzungen 1974). I can not find an online source for the latter but earlier work from Schmetterer already deals with generalizing for different loss functions than quadratic (I haven't read it to see if something similar as the proposition occurs in it)

### Uniformly Minimum Risk Unbiased Estimator

Another term that has been used is Uniformly Minimum Risk Unbiased Estimator (UMRUE)

see:

Qiguang, Wu. "Existence of the uniformly minimum risk unbiased estimator in seemingly unrelated regression system." Acta Mathematica Sinica 11.1 (1995): 23-28.

• Most estimation situations do not allow for an unbiased estimator. Jan 9 '19 at 15:04
• @Xi'an I guess this is close to the best you can get. Whuber's comment shows that you can not obtain a Uniformly Minimum Risk Biased Estimator since you can construct (non symmetric) loss functions that favor one type of bias over the other. Jan 9 '19 at 16:19
• @Xi'an this case of unbiased estimators occurs in another question by Richard Hardy. Indeed when the deterministic part is incorrect (and similar issues arise when the random part is incorrect), thus the model is biased, then the best estimator may not need to reflect the optimization of the variance due to the random part, but instead the optimization of the variance (or other loss function) due to the error of using biased models. Jan 11 '19 at 11:19
• I had accepted the answer prematurely, I think, as I was not really looking for a special case of unbiased estimators. I have unaccepted it now, but to compensate for the lost points, I have upvoted some of your other answers. Oct 22 '19 at 19:09
• @RichardHardy no problems. However with biased estimators you are generally not gonna get an estimator that optimises all loss functions. Just imagine loss functions that favours high/low errors which will favour different estimators with high/low bias. You can easily demonstrate it for a Bernoulli distributed variable where your sample is just size one. Oct 23 '19 at 7:32