21
$\begingroup$

In classical statistics, there is a definition that a statistic $T$ of a set of data $y_1, \ldots, y_n$ is defined to be complete for a parameter $\theta$ it is impossible to form an unbiased estimator of $0$ from it nontrivially. That is, the only way to have $E h(T (y )) = 0$ for all $\theta$ is to have $h$ be $0$ almost surely.

Is there a intuition behind this? Is seems like a rather mechanical way of defining this, I am aware this has been asked before, but was wondering if there was a very easy to understand intuition that would make introductory students have an easier time digesting the material.

$\endgroup$
  • 2
    $\begingroup$ That is a very good question, I had to dig into it myself. It turns out that the reason it is such a mechanical definition and does not appear intuitively meaningful to a standard practicioner like me is that it is primarily used for proving fundamental contributions in mathematical statistics. In particular, my short search revealed that the Lehmann-Scheffé theorem and Basu's theorem require completeness of a statistic in order to hold. These are contributions of the mid 1950s. I cannot offer you an intuitive explanation - but if you really want to build one, maybe the proofs associ $\endgroup$ – Jeremias K Feb 15 '16 at 21:43
18
$\begingroup$

I will try to add to the other answer. First, completeness is a technical condition which mainly is justified by the theorems that use it. So let us start with some related concepts and theorems where they occur.

Let $X=(X_1,X_2,\dotsc,X_n)$ represent a vector of iid data, which we model as having a distribution $f(x;\theta), \theta \in \Theta$ where the parameter $\theta$ governing the data is unknown. $T=T(X)$ is sufficient if the conditional distribution of $X \mid T$ does not depend on the parameter $\theta$. $V=V(X)$ is ancillary if the distribution of $V$ does not depend on $\theta$ (within the family $f(x;\theta)$). $U=U(X)$ is an unbiased estimator of zero if its expectation is zero, irrespective of $\theta$. $S=S(X)$ is a complete statistic if any unbiased estimator of zero based on $S$ is identically zero, that is, if $\DeclareMathOperator{\E}{\mathbb{E}} \E g(S)=0 (\text{for all $\theta$})$ then $g(S)=0$ a.e. (for all $\theta$).

Now, suppose you have two different unbiased estimators of $\theta$ based on the sufficient statistic $T$, $g_1(T), g_2(T)$. That is, in symbols $$ \E g_1(T)=\theta ,\\ \E g_2(T)=\theta $$ and $\DeclareMathOperator{\P}{\mathbb{P}} \P(g_1(T) \not= g_2(T) ) > 0$ (for all $\theta$). Then $g_1(T)-g_2(T)$ is an unbiased estimator of zero, which is not identically zero, proving that $T$ is not complete. So, completeness of an sufficient statistic $T$ gives us that there does exist only one unique unbiased estimator of $\theta$ based on $T$. That is already very close to the Lehmann–Scheffé theorem.

Let us look at some examples. Suppose $X_1, \dotsc, X_n$ now are iid uniform on the interval $(\theta, \theta+1)$. We can show that ($X_{(1)} < X_{(2)} < \dotsm < X_{(n)}$ is the order statistics) the pair $(X_{(1)}, X_{(n)})$ is sufficient, but it is not complete, because the difference $X_{(n)}-X_{(1)}$ is ancillary, we can compute its expectation, let it be $c$ (which is a function of $n$ only), and then $X_{(n)}-X_{(1)} -c$ will be an unbiased estimator of zero which is not identically zero. So our sufficient statistic, in this case, is not complete and sufficient. And we can see what that means: there exist functions of the sufficient statistic which are not informative about $\theta$ (in the context of the model). This cannot happen with a complete sufficient statistic; it is in a sense maximally informative, in that no functions of it are uninformative. On the other hand, if there is some function of the minimally sufficient statistic that has expectation zero, that could be seen as a noise term, disturbance/noise terms in models have expectation zero. So we could say that non-complete sufficient statistics do contain some noise.

Look again at the range $R=X_{(n)}-X_{(1)}$ in this example. Since its distribution does not depend on $\theta$, it doesn't by itself alone contain any information about $\theta$. But, together with the sufficient statistic, it does! How? Look at the case where $R=1$ is observed.Then, in the context of our (known to be true) model, we have perfect knowledge of $\theta$! Namely, we can say with certainty that $\theta = X_{(1)}$. You can check that any other value for $\theta$ then leads to either $X_{(1)}$ or $X_{(n)}$ being an impossible observation, under the assumed model. On the other hand, if we observe $R=0.1$, then the range of possible values for $\theta$ is rather large (exercise ...).

In this sense, the ancillary statistic $R$ does contain some information about the precision with which we can estimate $\theta$ based on this data and model. In this example, and others, the ancillary statistic $R$ "takes over the role of the sample size". Usually, confidence intervals and such needs the sample size $n$, but in this example, we can make a conditional confidence interval this is computed using only $R$, not $n$ (exercise.) This was an idea of Fisher, that inference should be conditional on some ancillary statistic.

Now, Basu's theorem: If $T$ is complete sufficient, then it is independent of any ancillary statistic. That is, inference based on a complete sufficient statistic is simpler, in that we do not need to consider conditional inference. Conditioning on a statistic which is independent of $T$ does not change anything, of course.

Then, a last example to give some more intuition. Change our uniform distribution example to a uniform distribution on the interval $(\theta_1, \theta_2)$ (with $\theta_1<\theta_2$). In this case the statistic $(X_{(1)}, X_{(n)})$ is complete and sufficient. What changed? We can see that completeness is really a property of the model. In the former case, we had a restricted parameter space. This restriction destroyed completeness by introducing relationships on the order statistics. By removing this restriction we got completeness! So, in a sense, lack of completeness means that the parameter space is not big enough, and by enlarging it we can hope to restore completeness (and thus, easier inference).

Some other examples where lack of completeness is caused by restrictions on the parameter space,

  • see my answer to: What kind of information is Fisher information?

  • Let $X_1, \dotsc, X_n$ be iid $\mathcal{Cauchy}(\theta,\sigma)$ (a location-scale model). Then the order statistics in sufficient but not complete. But now enlarge this model to a fully nonparametric model, still iid but from some completely unspecified distribution $F$. Then the order statistics is sufficient and complete.

  • For exponential families with canonical parameter space (that is, as large as possible) the minimal sufficient statistic is also complete. But in many cases, introducing restrictions on the parameter space, as with curved exponential families, destroys completeness.

A very relevant paper is An Interpretation of Completeness and Basu's Theorem.

$\endgroup$
7
$\begingroup$

Some intuition may be available from the theory of best (minimum variance) unbiased estimators.

If $E_\theta W=\tau(\theta)$ then $W$ is a best unbiased estimator of $\tau(\theta)$ iff $W$ is uncorrelated with all unbiased estimators of zero.

Proof: Let $W$ be an unbiased estimator uncorrelated with all unbiased estimators of zero. Let $W'$ be another estimator such that $E_\theta W'=E_\theta W=\tau(\theta)$. Write $W'=W+(W'-W)$. By assumption, $Var_\theta W'=Var_\theta W+Var_\theta (W'-W)$. Hence, for any $W'$, $Var_\theta W'\geq Var_\theta W$.

Now assume that $W$ is a best unbiased estimator. Let there be some other estimator $U$ with $E_\theta U=0$. $\phi_a:=W+aU$ is also unbiased for $\tau(\theta)$. We have $$Var_\theta \phi_a:=Var_\theta W+2aCov_\theta(W,U)+a^2Var_\theta U.$$ If there were a $\theta_0\in\Theta$ such that $Cov_{\theta_0}(W,U)<0$, we would obtain $Var_\theta \phi_a<Var_\theta W$ for $a\in(0,-2Cov_{\theta_0}(W,U)/Var_{\theta_0} U)$. $W$ could then not be the best unbiased estimator. QED

Intuitively, the result says that if an estimator is optimal, it must not be possible to improve it by just adding some noise to it, in the sense of combining it with an estimator that is just zero on average (being an unbiased estimator of zero).

Unfortunately, it is difficult to characterize all unbiased estimators of zero. The situation becomes much simpler if zero itself is the only unbiased estimator of zero, as any statistic $W$ satisfies $Cov_\theta(W,0)=0$. Completeness describes such a situation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.