Meaning of completeness of a statistic? [duplicate]

Question

From Wikipedia:

The statistic $s$ is said to be complete for the distribution of $X$ if for every measurable function $g$ (which must be independent of parameter $θ$) the following implication holds: $$ \mathbb{E}_\theta[g(s(X))] = 0, \forallθ \text{ implies that }P_θ(g(s(X)) = 0) = 1, \forall θ. $$ The statistic $s$ is said to be boundedly complete if the implication holds for all bounded functions $g$.

I read and agree with Xi'an and phaneron that a complete statistic means that "there can only be one unbiased estimator based on it".

1. But I don't understand what Wikipedia says at the beginning of the same article:

   In essence, it (completeness is a property of a statistic) is a condition which ensures that the parameters of the probability distribution representing the model can all be estimated on the basis of the statistic: it ensures that the distributions corresponding to different values of the parameters are distinct.

   • In what sense (and why) does completeness "ensures that the distributions corresponding to different values of the parameters are distinct"? is "the distributions" the distributions of a complete statistic?

   • In what sense (and why) does completeness "ensures that the parameters of the probability distribution representing the model can all be estimated on the basis of the statistic"?

2. [Optional: What does "bounded completeness" mean, compared to completeness?]

Answer 1

This is a very good question and one I've struggled with for quite some time. Here's how I've decided to think about it:

Take the contrapositive of the definition as stated in Wikipedia (which doesn't change the logical meaning at all):

\begin{align} {\rm If}\quad &\neg\ \forall \theta\ P(g(T(x))=0)=1 \\ {\rm then}\quad &\neg\ \forall \theta\ E(g(T(x))) = 0 \end{align}

In other words, if there is a parameter value such that $g(T(x))$ is not almost surely $0$, then there is a parameter value such that the expected value of that statistic is not $0$.

Hmm. What does that even mean?

Let's ask what happens when $T(x)$ is NOT complete...

A statistic $T(x)$ that is NOT complete will have at least one parameter value such that $g(T(x))$ is not almost surely $0$ for that value, and yet it's expected value is $0$ for all parameter values (including this one).

In other words, there are values of $\theta$ for which $g(T(x))$ has a non-trivial distribution around it (it has some random variation in it), and yet the expected value of $g(T(x))$ is nonetheless always $0$--it doesn't budge, no matter how much $\theta$ is different.

A complete statistic, on the other have, will budge in it's expected value eventually if $g(T(x))$ is non-trivially distributed and centered at $0$ for some $\theta$.

Put another way, if we find a function $g(\cdot)$ where the expected value is zero for some $\theta$ value (say $\theta_0$) and it has a non-trivial distribution given that value of $\theta$, then there must be another value of $\theta$ out there (say, $\theta_1 \ne \theta_0$) that results in a different expectation for $g(T(x))$.

This means we can actually use this statistic for hypothesis testing and informative estimation in the context of an assumed distribution for our data. We want to be able to center it around a hypothesized value of $\theta$ and get it to have expectation 0 for that hypothesized value of $\theta$, but not for all other values of $\theta$. But if the statistic is not complete we may not be able to do this: we may be unable to reject any hypothesized values of $\theta$. But then we can't build confidence intervals and do statistical estimation.

Answer 2

Geometrically, completeness means something like this: if a vector $g(T)$ is orthogonal to the p.d.f. $f_\theta$ of $T$ for each $\theta$, $$\mathbb E_\theta g(T) = \langle g(T),f_\theta\rangle=0$$ then $g(T)=0$ i.e., the functions $f_\theta$ for varying $\theta$ span the whole space of functions of $T$. So in a way it would be more natural to say that

$\theta$ is complete for $T$

than what we do say,

$T$ is complete for $\theta$.

This way it is not so strange that a constant function would be "complete"!

---

Maybe an example helps.

Suppose $X$ and $Y$ are independent and identically distributed Bernoulli($\theta$) random variables taking values in $\{0,1\}$, and $Z=X-Y$. Then $Z$ is incomplete for $\theta$, because taking $g=\text{identity}$, $$\mathbb E_\theta(Z)=0$$ for all $0<\theta<1$, but nevertheless $\mathbb P_\theta(Z=0)\ne 1$.

Answer 3

I found this very helpful:

Definition: A statistic $T$ is called complete if $E_\theta[g(T)] = 0$ for all $\theta$ and some function $g$ implies that $P_\theta(g(T) = 0) = 1$ for all $\theta$.

Think of this as analog to vectors and whether or not the vectors {$v_1, \ldots , v_n$} form a complete set (=basis) of the vector space.

• If they span the whole space, any $v$ can be written as a linear combination of these vectors: $v = \sum_j a_j \cdot v_j$.
• Furthermore, if a vector $w$ is orthogonal to all $v_j$’s, then $w = 0$.

To make the connection with the completeness definition, let’s consider the case of a discrete probability distribution. We start by writing out the completeness condition $$ 0 = E_\theta[g(t)] = \sum_t g(t) \cdot P_\theta(T = t) = \sum_j g(t_j) \cdot P_\theta(T = t_j) = \begin{bmatrix} g(t_1)\\ g(t_2)\\ \ldots \\ g(t_n)\\ \end{bmatrix} \cdot \begin{bmatrix} p_\theta(t_1)\\ p_\theta(t_2)\\ \ldots \\ p_\theta(t_n)\\ \end{bmatrix} $$ for all $\theta$. Here we expressed the sum as a scalar product of two vectors $$(g(t_1), g(t_2),...)$$ and $$(p_\theta(t_1), p_\theta(t_2),...)$$, with $p_\theta(t_j) = P_\theta(T = t_j) \ne 0$ -- we consider only positive probabilities, because if $p(t_j) = 0$ this does not tell us anything about the function $g(t_j)$. Now we see the analogy to the orthogonality condition discussed above.

In principle it could be that the $g(t_j)$'s are non-zero, but that they sum to zero. However, as stated by Lhunt, this is only possible if

• the probability vector $(p_\theta(t_1), p_\theta(t_2),...)$ does either not change at all if $\theta$ is varied,
• or if it changes in a "simple way", e.g. it jumps from one value for all $j$'s to an other value for all $j$'s,
• or if it changes in a "correlated way", which would be a nightmare to handle.

Thus, the cross-cancelation of terms is only possible if the probability distribution provides either a "boring set of basis vectors" or a nightmare.

In contrast, if the probability distribution provides a "sufficient rich set of basis vectors", the equation for the expectation value implies $g(t_j) = 0$ almost everywhere. By almost everywhere we mean, that there could be a set of probability zero, where $g(t_j) \ne 0$ -- e.g. in the case of a continuous probability distribution this could be a set of single points.

We also see that the terminology is somewhat misleading. It would be more accurate to call the family of distributions $p_\theta(\cdot)$ complete (rather than the statistic $T$) -- as stated in the original question. In any event, completeness means that the collection of distributions for all possible values of $\theta$ provides a sufficiently rich set of vectors.