Best unbiased estimator for a location family

Question

Mainly for pedagogical reasons, I'm considering the "simple" one dimensional model: $$x=\theta+\epsilon$$ where $\epsilon$ has a known distribution $p$ that is independent of $\theta$. This distribution can be anything, just assuming some regularity conditions, compact support... something that leaves a lot of freedom for the choice of $p$. Somehow I'm looking for properties that hold for "almost any $p$", ignoring special cases with strong underlying algebraic properties (like in an exponential family).

Consider an independent sample $X=(x_1,x_2,x_3,...,x_n)$.

I'm looking for a general way to construct the best unbiased estimator if it exist. Not asymptotically I mean but for fixed finite $n$. It happens to be a bit more than difficult I thought. The questions I face are:

• what is a minimal sufficient statistics (is it just reordered $X$)?
• the MLE has a constant bias (see note), is debiased MLE the best unbiased estimator?
• how to use an ancillary statistics to find the best unbiased estimator? Would it yield the same estimator?

I guess I'm not the first person on earth to ask myself these questions. Have you done this kind of analysis, do you know results, the "right" method, or do you know of an article that studied something like it?

Note: proof that MLE $\hat\theta$ has constant bias:

$E_\theta(\hat\theta)=E_\theta(\arg\max_t \prod_ip(x_i-t))=E_\theta(\arg\max_t \prod_ip(\epsilon_i-(t-\theta)))\\=E_\theta(\arg\max_u \prod_ip(\epsilon_i-u)+\theta)=E_0(\hat\theta)+\theta=c+\theta $

When $p$ is symetric around 0, this bias is 0: since $-\epsilon$ has the same distribution as $\epsilon$: $E_0 (\hat\theta)=E_\theta(\arg\max_t \prod_ip(-\epsilon_i-t))\\=E_\theta(\arg\max_t \prod_ip(\epsilon_i+t))=E_\theta(-\arg\max_u \prod_ip(\epsilon_i-u))=-E_0(\hat\theta)$

Answer 1

First, if the distribution $p$ is unspecified (or does not belong to an extended exponential family as, e.g., the location Cauchy distributions), the order statistic is indeed the minimal sufficient statistic. (See my answer to an earlier X validated question and Lehmann and Casella (1998).)

As pointed out in an answer to an earlier X'ed question, there is no reason for the MLE of the parameter $\theta$ to be unbiased [except, as updated by the OP, when the density $p$ is symmetric]. There is further no reason for the bias to be constant for all $\theta$'s across all $p$'s [if constant for a given $p$]. And there is further² no reason for an UNMVUE to exist. Actually, since the order statistic is minimal sufficient, it cannot be complete: $X_{(i)}-X_{(j)}$ is for instance ancillary $(i\ne j)$, hence does not allow for Lehmann-Scheffé to apply. (Although there also exist settings when the UNMVUE exists while a complete sufficient statistic does not.)

An example taken from Lehmann (1983, p.76) goes as follows: take $X$ with support $\{-1,0,1,\ldots\}$ with probabilities $(0<p<1)$ $$\mathbb{P}(X=-1)=p,\quad\mathbb{P}(X=k)=(1-p)^2p^k\quad > k=0,1,\ldots$$ Finding the minimum variance unbiased estimator of $p$ for $p=p_0$ amounts to minimising$$\sum_{k=-1}^\infty > \mathbb{P}(X=k)[\mathbb{I}_{-1}(k)-ak]^2$$which returns the solution $$a^\star=-p_0\Bigg/\left[ p_0+(1-p_0)^2\sum_{k=1}^\infty k^2p_0^k > \right]$$ which depends on $p_0$, hence prohibits the existence of a uniformly minimum variance unbiased estimator.

In the case of a location parameter $\theta$, Lehmann and Scheffé (1950) showed that the Uniform $\mathcal{U}[\theta-1,\theta+1]$ does not allow for an UMVUE, apart from constant functions of $\theta$. (This perfectly fits the setting of the question.)

Most interestingly, when the distribution $p$ is unknown (with mean zero), $\bar{X}_n$ is the best unbiased estimator, as stated in Lehmann (1983):

To answer more specifically the question about the (possibly recentred) MLE, it should be compared to the Pitman best equivariant estimator, which dominates the original (and equivariant) MLE and the recentred and also equivariant MLE, in terms of mean square error. As later pointed out by the OP in a comment [and also found in Lehmann (1983, Lemma 3.1.3)], the Pitman best equivariant estimator of the location parameter is always unbiased and hence dominates the recentred MLE. However, in the specific case of the Cauchy location parameter, there exist no UMVUE for the location parameter. This completes a paper by Bondesson (1975) where he proves the non-existence of a UMV-estimator of θ, provided that the tail of the density tends to zero rapidly enough.

Answer 2

As pointed by Xi'an, an example where no best unbiased estimator exists is $x\sim\mathcal{U}[θ−1,θ+1]$. The proof is very technical and it might be interesting to extend it to "almost any $p$" which may be very difficult.

However the model here has a strong invariance property under translation. It makes sense looking only at equivariant estimators $S$ who are those such as:

$$S(X+(t,t,....t))=S(X)+t$$

Intuitively: when you shift the whole data, you shift the estimator just the same which sounds pretty natural. While there may be no best unbiased estimator, there is a unique best equivariant estimator.

Equivariant estimators have constant MSE simply because their distribution just shifts with $\theta$. As a consequence they can be compared with a total order and you can simply evaluate their MSE under $\theta=0$. Note that they all have constant bias and can always be recentred to improve them. Thus the best equivariant estimator is necesserely unbiased.

To construct the best equivariant estimator, just write the ancillary statistics $\Delta=(X_2-X_1,X_3-X_1,...,X_n-X_1)$. It is easy to see that the equivariant estimators are those such as:

$$S(X)=X_1-f(\Delta)$$

Now the best $f$ is the one such as the MSE is the smallest (for example under $\theta=0$), it minimize $E_0((X_1-f(\Delta)^2)$ and the unique solution is $f(\Delta)=E_0(X_1|\Delta)$ : the projection of $X_1$ onto the space of $\Delta$-measurable variables. Note that other ancillary statistics could be used and would yield the same estimator.

Thus the best equivariant estimator is: $$\hat\theta = X_1-E_0(X_1|\Delta)$$

For the general theory of equivariant estimators see: Invariant estimation. This estimator is known as the Pitman equivariant estimator.

Other equivariant estimators include:

• recentered (debiased) MLE
• Bayesian $E(\theta|X)$ with uniform prior on $\theta$

We can be sure they are not better. But in fact our Pitman estimator is the same as the Bayesian one (see the reference). MLE however is generally less good. See Xi'an answer for an example for the Cauchy distribution.