What is the intuition behind defining completeness in a statistic as being impossible to form an unbiased estimator of $0$ from it?

Question

In classical statistics, there is a definition that a statistic $T$ of a set of data $y_1, \ldots, y_n$ is defined to be complete for a parameter $\theta$ if it is impossible to form an unbiased estimator of $0$ from it nontrivially. That is, the only way to have $E h(T (y )) = 0$ for all $\theta$ is to have $h$ be $0$ almost surely.  

Is there a intuition behind this? Is seems like a rather mechanical way of defining this, I am aware this has been asked before, but was wondering if there was a very easy to understand intuition that would make introductory students have an easier time digesting the material.

Answer 1

I will try to add to the other answer. First, completeness is a technical condition which is justified mainly by the theorems that use it. So let us start with some related concepts and theorems where they occur.

Let $X=(X_1,X_2,\dotsc,X_n)$ represent a vector of iid data, which we model as having a distribution $f(x;\theta), \theta \in \Theta$ where the parameter $\theta$ governing the data is unknown. $T=T(X)$ is sufficient if the conditional distribution of $X \mid T$ does not depend on the parameter $\theta$. $V=V(X)$ is ancillary if the distribution of $V$ does not depend on $\theta$ (within the family $f(x;\theta)$). $U=U(X)$ is an unbiased estimator of zero if its expectation is zero, irrespective of $\theta$. $S=S(X)$ is a complete statistic if any unbiased estimator of zero based on $S$ is identically zero, that is, if $\DeclareMathOperator{\E}{\mathbb{E}} \E g(S)=0 (\text{for all $\theta$})$ then $g(S)=0$ a.e. (for all $\theta$).

Now, suppose you have two different unbiased estimators of $\theta$ based on the sufficient statistic $T$, $g_1(T), g_2(T)$. That is, in symbols $$ \E g_1(T)=\theta ,\\ \E g_2(T)=\theta $$ and $\DeclareMathOperator{\P}{\mathbb{P}} \P(g_1(T) \not= g_2(T) ) > 0$ (for all $\theta$). Then $g_1(T)-g_2(T)$ is an unbiased estimator of zero, which is not identically zero, proving that $T$ is not complete. So, completeness of an sufficient statistic $T$ gives us that there exists only one unique unbiased estimator of $\theta$ based on $T$. That is already very close to the Lehmann–Scheffé theorem.

Let us look at some examples. Suppose $X_1, \dotsc, X_n$ now are iid uniform on the interval $(\theta, \theta+1)$. We can show that ($X_{(1)} < X_{(2)} < \dotsm < X_{(n)}$ are the order statistics) the pair $(X_{(1)}, X_{(n)})$ is sufficient, but it is not complete, because the difference $X_{(n)}-X_{(1)}$ is ancillary; we can compute its expectation, let it be $c$ (which is a function of $n$ only), and then $X_{(n)}-X_{(1)} -c$ will be an unbiased estimator of zero which is not identically zero. So our sufficient statistic, in this case, is not complete and sufficient. And we can see what that means: there exist functions of the sufficient statistic which are not informative about $\theta$ (in the context of the model). This cannot happen with a complete sufficient statistic; it is in a sense maximally informative, in that no functions of it are uninformative. On the other hand, if there is some function of the minimally sufficient statistic that has expectation zero, that could be seen as a noise term; disturbance/noise terms in models have expectation zero. So we could say that non-complete sufficient statistics do contain some noise.

Look again at the range $R=X_{(n)}-X_{(1)}$ in this example. Since its distribution does not depend on $\theta$, it doesn't by itself alone contain any information about $\theta$. But, together with the sufficient statistic, it does! How? Look at the case where $R=1$ is observed.Then, in the context of our (known to be true) model, we have perfect knowledge of $\theta$! Namely, we can say with certainty that $\theta = X_{(1)}$. You can check that any other value for $\theta$ then leads to either $X_{(1)}$ or $X_{(n)}$ being an impossible observation, under the assumed model. On the other hand, if we observe $R=0.1$, then the range of possible values for $\theta$ is rather large (exercise ...).

In this sense, the ancillary statistic $R$ does contain some information about the precision with which we can estimate $\theta$ based on this data and model. In this example, and others, the ancillary statistic $R$ "takes over the role of the sample size". Usually, confidence intervals and such need the sample size $n$, but in this example, we can make a conditional confidence interval this is computed using only $R$, not $n$ (exercise.) This was an idea of Fisher, that inference should be conditional on some ancillary statistic.

Now, Basu's theorem: If $T$ is complete sufficient, then it is independent of any ancillary statistic. That is, inference based on a complete sufficient statistic is simpler, in that we do not need to consider conditional inference. Conditioning on a statistic which is independent of $T$ does not change anything, of course.

Then, a last example to give some more intuition. Change our uniform distribution example to a uniform distribution on the interval $(\theta_1, \theta_2)$ (with $\theta_1<\theta_2$). In this case the statistic $(X_{(1)}, X_{(n)})$ is complete and sufficient. What changed? We can see that completeness is really a property of the model. In the former case, we had a restricted parameter space. This restriction destroyed completeness by introducing relationships on the order statistics. By removing this restriction we got completeness! So, in a sense, lack of completeness means that the parameter space is not big enough, and by enlarging it we can hope to restore completeness (and thus, easier inference).

Some other examples where lack of completeness is caused by restrictions on the parameter space,

• see my answer to: What kind of information is Fisher information?

• Let $X_1, \dotsc, X_n$ be iid $\mathcal{Cauchy}(\theta,\sigma)$ (a location-scale model). Then the order statistics are sufficient but not complete. But now enlarge this model to a fully nonparametric model, still iid but from some completely unspecified distribution $F$. Then the order statistics are sufficient and complete.

• For exponential families with canonical parameter space (that is, as large as possible) the minimal sufficient statistic is also complete. But in many cases, introducing restrictions on the parameter space, as with curved exponential families, destroys completeness.

A very relevant paper is Lehmann (1981), J. Am. Stat. Assoc., 76, 374, "An Interpretation of Completeness and Basu's Theorem".

Answer 2

Some intuition may be available from the theory of best (minimum variance) unbiased estimators (Casella and Berger's Statistical Inference (2002), Theorem 7.3.20).

If $\operatorname{E}_\theta W=\tau(\theta)$ then $W$ is a best unbiased estimator of $\tau(\theta)$ iff $W$ is uncorrelated with all other unbiased estimators of zero.

Proof: Let $W$ be an unbiased estimator uncorrelated with all unbiased estimators of zero. Let $W'$ be another estimator such that $\operatorname{E}_\theta W'=\operatorname{E}_\theta W=\tau(\theta)$. Write $W'=W+(W'-W)$. By assumption, $\operatorname{Var}_\theta W'=\operatorname{Var}_\theta W+\operatorname{Var}_\theta (W'-W)$. Hence, for any $W'$, $\operatorname{Var}_\theta W'\geq \operatorname{Var}_\theta W$.

Now assume that $W$ is a best unbiased estimator. Let there be some other estimator $U$ with $\operatorname{E}_\theta U=0$. $\phi_a:=W+aU$ is also unbiased for $\tau(\theta)$. We have $$\operatorname{Var}_\theta \phi_a := \operatorname{Var}_\theta W + 2a \operatorname{Cov}_\theta(W,U) + a^2 \operatorname{Var}_\theta U.$$ If there were a $\theta_0\in\Theta$ such that $\operatorname{Cov}_{\theta_0}(W,U)<0$, we would obtain $\operatorname{Var}_\theta \phi_a < \operatorname{Var}_\theta W$ for $a\in(0,-2 \operatorname{Cov}_{\theta_0}(W,U) / \operatorname{Var}_{\theta_0} U)$. $W$ could then not be the best unbiased estimator. QED

Intuitively, the result says that if an estimator is optimal, it must not be possible to improve it by just adding some noise to it, in the sense of combining it with an estimator that is just zero on average (being an unbiased estimator of zero).

Unfortunately, it is difficult to characterize all unbiased estimators of zero. The situation becomes much simpler if zero itself is the only unbiased estimator of zero, as any statistic $W$ satisfies $\operatorname{Cov}_\theta(W,0)=0$. Completeness describes such a situation.

Answer 3

The key point about estimators of zero (as Christoph Hanck says above) is that they are differences between unbiased estimators. If $U$ and $V$ are two unbiased estimators of $\theta$ then $Z=U-V$ is an unbiased estimator of zero; conversely, if $V$ is unbiased for $\theta$ and $Z$ is unbiased for zero then $V+Z=U$ is unbiased for $\theta$.

So saying that there are no non-trivial unbiased estimators of zero, is just a way to say that there are no non-trivially-different unbiased estimators of $\theta$.

In general, there could be lots of unbiased estimators of any given $\theta$: for example, if $X_i\sim N(\mu,\sigma)$, then $\bar X$ or the median of $X$ or the midrange or $X_{17}$ are all unbiased estimators of $\theta$. However, if there is a minimal sufficient statistic $T$, you'd expect the best unbiased estimators to be functions of $T$ (either heuristically, or by the Rao-Blackwell theorem)

In the $N(\mu,\sigma)$ case, a minimal sufficient statistic is $\bar X$, and only one of the four unbiased estimators I mentioned is a function of it. In fact, there is no other unbiased estimator that's a function of $T$. It's unique!

Does this always happen? Sadly, no. But models and statistics where it does happen are interesting, and we'd expect the uniqueness to be useful in proving other results. If you want to show that something is the best estimator satisfying some condition it's very convenient for it to be the only estimator satisfying that condition.

We use the word complete to describe this setting, and it does indeed turn out to be technically useful, though only in a small number of interesting results.