23
$\begingroup$

I've just started studying statistics and I can't get an intuitive understanding of sufficiency. To be more precise I can't understand how to show that the following two paragraphs are equivalent:

Roughly, given a set X of independent identically distributed data conditioned on an unknown parameter θ, a sufficient statistic is a function T(X) whose value contains all the information needed to compute any estimate of the parameter.

A statistic T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic T(X), does not depend on the parameter θ.

(I've taken the quotes from Sufficient statistic)

Though I understand the second statement, and I can use the factorization theorem to show if a given statistic is sufficient, I can't understand why a statistic with such a property has also the property that it "contains all the information needed to compute any estimate of the parameter". I am not looking for a formal proof, which would help anyway to refine my understanding, I'd like to get an intuitive explanation of why the two statements are equivalent.

To recap, my questions are: why the two statements are equivalent? Could someone provide an intuitive explanation for their equivalence?

$\endgroup$
13
  • 2
    $\begingroup$ The main intuitive idea is that you sometimes don't need to see the whole sample because you can find a statistic that summarises all the information needed from the sample. Take, for example, a binomial distribution: all you need to know for your model is the sum of successes. You don't lose anything of value if I only tell you that $\sum_{i}^{n} x_i = c$, instead of showing you the whole set of sampled values $x = \{1, 0, 0, 1, 0, 1, ... \}$. $\endgroup$
    – mugen
    Commented Jan 18, 2015 at 23:40
  • 7
    $\begingroup$ Strictly speaking, the first quotation is just plain wrong. There are plenty of estimators that can be computed from the whole dataset which cannot be computed solely from sufficient statistics. That's one reason the quote begins "roughly." Another reason is that it doesn't supply a quantitative or rigorous definition of "information." Since a much more accurate (but still intuitive) characterization had been given in the preceding paragraph, though, there is little problem with this quotation in the proper context. $\endgroup$
    – whuber
    Commented Jan 19, 2015 at 1:52
  • 1
    $\begingroup$ It has connection to maximum likelihood and it is essentially the information needed in the maximum likelihood $\endgroup$
    – Kamster
    Commented Jan 19, 2015 at 2:56
  • 1
    $\begingroup$ Following the comments of whuber and @Kamster, I probably got a better understanding. When we say that a sufficient statistic contains all the information needed to compute any estimate of the parameter, do we actually mean that it is enough to compute the maximum likelihood estimator (which is a function of all sufficient statistics)? It this is true, the issue was all related to the (non-)definition of "information", as whuber suggested, and my question is answered. $\endgroup$
    – gcoll
    Commented Jan 19, 2015 at 22:07
  • 2
    $\begingroup$ Coincidentally we had been discussing about this just yesterday at Ten Fold @AdrianKeister. $\endgroup$ Commented Jun 14 at 18:19

7 Answers 7

7
$\begingroup$

As I was studying about sufficiency I came across your question because I also wanted to understand the intuition about From what I've gathered this is what I come up with (let me know what you think, if I made any mistakes, etc).

Let $X_1,\ldots,X_n$ be a random sample from a Poisson distribution with mean $\theta>0$.

We know that $T({\bf{X}})=\sum_{i=1}^{n} X_i$ is a sufficient statistic for $\theta$, since the conditional distribution of $X_1,\ldots,X_n$ given $T({\bf{X}})$ is free of $\theta$, in other words, does not depend on $\theta$.

Now, statistician $A$ knows that $X_1,\ldots,X_n \overset{i.i.d}{\sim} \operatorname{Poisson}(4)$ and creates $n=400$ random values from this distribution:

    n <- 400
    theta <- 4
    set.seed(1234)
    x <- rpois(n, theta)
    y=sum(x)
       
    freq.x <- table(x) # We will use this latter on
    rel.freq.x <- freq.x/sum(freq.x)

For the values statistician $A$ has created, he takes the sum of it and asks statistician $B$ the following:

"I have these sample values $x_1,\ldots,x_n$ taken from a Poisson distribution. Knowing that $\sum_{i=1}^{n} x_i = y = 4068$, what can you tell me about this distribution?"

So, knowing only that $\sum_{i=1}^{n} x_i = y = 4068$ (and the fact that the sample arose from a Poisson distribution) is sufficient for statistician $B$ to say anything about $\theta$? Since we know that this is a sufficient statistic we know that the answer is "yes".

To gain some intuition about the meaning of this, let's do the following (taken from Hogg & Mckean & Craig's "Introduction to Mathematical Statistics", 7th edition, exercise 7.1.9):

"$B$ decides to create some fake observations, which he calls $z_1,z_2,\ldots,z_n$ (as he knows they will probably not be equal the original $x$-values) as follows. He notes that the conditional probability of independent Poisson random variables $Z_1,Z_2\ldots,Z_n$ being equal to $z_1,z_2,\ldots,z_n$, given $\sum z_i = y$, is

$$\cfrac{\frac{\theta^{z_1}e^{-\theta}}{z_1!} \frac{\theta^{z_2}e^{-\theta}}{z_2!} \cdots \frac{\theta^{z_n}e^{-\theta}}{z_n!}}{\frac{n \theta^{y}e^{-n\theta}}{y!}}=\frac{y!}{z_1!z_2! \cdots z_n!} \left(\frac{1}{n}\right)^{z_1} \left(\frac{1}{n}\right)^{z_2} \cdots \left(\frac{1}{n}\right)^{z_n}$$

since $Y=\sum Z_i$ has a Poisson distribution with mean $n \theta$. The latter distribution is multinomial with $y$ independent trials, each terminating in one of $n$ mutually exclusive and exhaustive ways, each of which has the same probability $1/n$. Accordingly, $B$ runs such a multinomial experiment $y$ independent trials and obtains $z_1,\ldots,z_n$."

This is what the exercise states. So, let's do exactly that:

    # Fake observations from multinomial experiment
    prob <- rep(1/n, n)
    set.seed(1234)
    z <- as.numeric(t(rmultinom(y, n=c(1:n), prob)))
    y.fake <- sum(z) # y and y.fake must be equal
    freq.z <- table(z)
    rel.freq.z <- freq.z/sum(freq.z)

And let's see what $Z$ looks like (I'm also plotting the real density of Poisson(4) for $k=0,1,\ldots,13$ - anything above 13 is pratically zero -, for comparison):

    # Verifying distributions
    k <- 13
    plot(x=c(0:k), y=dpois(c(0:k), lambda=theta, 
         log = FALSE),t="o",ylab="Probability",xlab="k",
         xlim=c(0,k), ylim=c(0, max(c(rel.freq.x, rel.freq.z))))
    lines(rel.freq.z, t="o", col="green", pch=4)
    legend(8,0.2, legend=c("Real Poisson", "Random Z given y"), 
           col = c("black", "green"), pch=c(1,4))

enter image description here

So, knowing nothing about $\theta$ and knowing only the sufficient statistic $Y=\sum X_i$ we were able to recreate a "distribution" that looks a lot like a Poisson(4) distribution (as $n$ increases, the two curves become more similar).

Now, comparing $X$ and $Z|y$:

    plot(rel.freq.x, t="o", pch=16, col="red", 
         ylab="Relative Frequency", xlab="k",
         ylim=c(0, max(c(rel.freq.x, rel.freq.z))))
    lines(rel.freq.z, t="o", col="green", pch=4)
    legend(7, 0.2, legend=c("Random X", "Random Z given y"), 
           col = c("red", "green"), pch=c(16,4))

enter image description here

We see that they are pretty similar, as well (as expected)

So, "for the purpose of making a statistical decision, we can ignore the individual random variables $X_i$ and base the decision entirely on the $Y=X_1+X_2+\cdots+X_n$" (Ash, R. "Statistical Inference: A concise course", page 59).

$\endgroup$
5
$\begingroup$

Following the comments of @whuber and @Kamster, I probably got a better understanding. When we say that a sufficient statistic contains all the information needed to compute any estimate of the parameter, what we actually mean is that it is enough to compute the maximum likelihood estimator (which is a function of all sufficient statistics).

Given that I am answering my own question, and so I am not 100% sure of the answer, I will not mark it as correct until I get some feedback. Please add any comment and down-vote if you think I am being wrong/imprecise/etc...

(Let me know if this is not compatible with SE etiquette, being this my first question I beg your clemency if I am violating any rule)

$\endgroup$
4
+50
$\begingroup$

First of all, the following phrase from the first paragraph is a bit awkward. (it seems to originate from Wikipedia 23 september 2010, and with Google you can find several copies in articles and thesis reports)

contains all the information needed to compute any estimate of the parameter.

  • The sufficient statistic does not contain all the information. It is the sufficient statistic plus the (assumed) knowledge about the data generating process.

    (the point of statistics other than the sufficient statistic, is that the information might be incorrect, and what is assumed to be the sufficient statistic may not perform well)

  • The sufficient statistic can not be used to reconstruct the original data. Thus, not all statistics and estimates can be computed.

    (for example, for a normal distribution with unknown mean, the first data point is a possible estimate of the location parameter, and can not be computed based on solely the sufficient statistic, the a sample mean)

    (however, as explained below we can stimulate those statistics and get the same performance if the assumptions about the model are true)

Factorisation theorem and conditional probability

One way to regard the factorisation theorem, is that it considers some data generating process as effectively equivalent to a two step process.

  • In the first step we generate the sufficient statistic(s), which is a random process that depends on the parameters of interest.
  • In the second step we generate the data based on the sufficient statistic only.

This second step doesn't add any more information about the distribution parameter. In the example below it is just a random distribution of points on a n-sphere or a random permutation, and it is independent of the distribution parameter.

Related is an answer in this question Need help understanding Sufficient Statistics and using the formal definition

  • If we can factorise the distribution

    $$f_\mathbf x(\mathbf x\mid\theta) = g(t;\theta)\cdot h(\mathbf x)$$

  • Then we can also write it in terms of the data being conditional on the sufficient statistic (and only indirectly dependent on the parameter $\theta$)

    $$f_\mathbf x(\mathbf x\mid\theta) = f_T(t,\theta) \cdot f_\mathbf x(\mathbf x\mid T=t)$$

Example Normal distribution

Consider the generation of IID normal distributed variables with known mean and unknown variance $$X_i \sim N(0,\sigma^2)$$ they are equivalent to a Box-Muller transform or a n-dimensional generalisation (How to use Box-Muller transform to generate n-dimensional normal random variables). We can sample $Y_i$ uniform on an n-1 sphere, and scale by a radius that is gamma distributed.

$$X_i = Y_i \cdot \sqrt{R} \\ \text{with} \\ R \sim \Gamma \left( \frac{n}{2},2\sigma \right) \\ Y_i \sim \text{uniform on an}~ n-1 ~\text{sphere}$$

The $Y_i$ are independent from $\sigma$ and have no information about $\sigma$.

Example Bernoulli variables

The comment by mugen gives an example with Bernoulli variables. The variables $$X_i \sim \operatorname{Bernoulli}(p)$$ can be alternatively generated by two steps.

  1. First the total counts (which depends on $p$) $$K \sim \text{Binom}(n,p)$$
  2. Second sample a permutation of the numbers $1,\dots,n$
    $$Y_1,Y_2, \dots , Y_n \sim \text{Permutation}(1:n)$$

and $$X_i = \mathbf{1}_{Y_i \leq K} = \begin{cases} 1 & \text{if $Y_i \leq K$} \\ 0 & \text{if $Y_i > K$} \\ \end{cases}$$

Computing any other statistic

The first paragraph is not completely senseless. I once came across the idea independently in the question Sufficient Statistic and Maximum likelihood, where I used the mean to compute a median. This was on response to a comment

I might be able to come up with a better estimator for $\theta$ which takes on a strange functional form, and maybe then the observed value of the sufficient statistic will not be enough to allow me to infer about $\theta$ in the way that I wish."

Whatever estimator one may come up with, it can be created out of the sufficient statistic as well.

(it is however a weird case, because when there is a sufficient statistic, then other strange estimators will not be better. Unless, the assumptions about the sufficient statistic are false. But, that's actually the same as saying that an estimator is better because the assumptions behind the sufficient statistic are wrong, in which case it makes no sense consider the sufficient statistic as containing all information to compute other statistics).

So, we can interpret

all the information needed to compute any estimate of the parameter.

as the following statement occuring in this answer

The inference with a method that is not based on the sufficient statistic can be turned into a method that is based on the sufficient statistic with equivalent performance.

If there is some estimate that is not directly computed from the sufficient statistic (e.g. an estimate based on the mean, when we have a uniform distributed sample $U(0,\theta)$ and the maximum of that sample), then we can use the sufficient statistic to generate surrogate data and compute that estimate. It will have the same performance because there is no difference in the sample distribution of the original data and the surrogate data.

My answer in that related question demonstrates it with normal distributed data and the computation of the median based on the sufficient statistic, the mean.

Relationship with efficiency instead of information or computing any estimate

Also helpful might be this comment from the chat

the way it appears to me, sufficiency (and efficiency) originated with Fisher by considering the question which statistic are the best to estimate a certain population parameter, and the realization that estimates based on a particular statistic can be improved upon by using another estimate. Those estimates for which this is not the case are sufficient statistics. Possibly this post helps.

Here the idea is not so much whether we can compute any other statistic based on the sufficient statistic, but more whether any other statistic can improve the estimate.

One can regard the distribution of one statistic conditional on another statistic, and consider whether it is dependent on the population parameter. For a sufficient statistic, the conditional distribution of any other statistic is independent from the distribution parameters. This relates to Rao-Blackwellisation.

$\endgroup$
17
  • $\begingroup$ @User1865345 I had been thinking something like that. One can regard something like a typical list like "the following are the same" and this question has two statements. The comment in that chat adds more like a third statement. I was considering adding that third statement, but I did not really wanted to make the post much longer (I find some of the other answers too long, but I am reading this using a small smartphone)... $\endgroup$ Commented Jun 16 at 8:43
  • $\begingroup$ ... The third statement would be along the lines that the density distribution of the data can be regarded as several isosurfaces and these are invariant to changes of the distribution parameter. (But this image is limited to that case in that early article by Fisher). The current answer here contains the same link as the comment. Possibly pivot variables might generalise this idea. $\endgroup$ Commented Jun 16 at 8:47
  • $\begingroup$ @User1865345 I have made a question regarding this view of the sufficient statistic stats.stackexchange.com/q/649322/164061 $\endgroup$ Commented Jun 16 at 9:48
  • 1
    $\begingroup$ So you need to generate the new data based on a multivariate normal distribution $$Y \sim N(\bar{X}, \Sigma)$$ with the covariance being $$\Sigma_{ij} = \begin{cases} 1-\frac{1}{n} & \quad {\text{if $i=j$}}\\ -\frac{1}{n} & \quad {\text{if $i\neq j$}}\\ \end{cases}$$ $\endgroup$ Commented Jun 17 at 13:57
  • 1
    $\begingroup$ +1 because this is the only (undeleted!) answer to mention what I believe is the crux of the matter, the Rao-Blackwell theorem. The first set of bullet points alone earned my upvote. $\endgroup$
    – whuber
    Commented Jun 18 at 19:52
3
$\begingroup$

To have at least one answer that says this explicitly: No, a sufficient statistic does not contain all the information to compute any estimator of the parameter, and not even all reasonable estimators.

There is some theory, as explained in other answers, that shows why estimators that have certain optimality properties including maximum likelihood can be written down as functions of a sufficient statistic only.

However there may be desirable characteristics of estimators that cannot be fulfilled by estimators that depend on the data through the sufficient statistic only.

A major criterion for a desirable characteristic of an estimator that is in conflict with sufficiency is the breakdown point (robustness): The arithmetic mean of the data is sufficient for estimating the Gaussian mean, but it is well known and very easy to see that by replacing even a single observation by an extreme outlier, the mean can be driven arbitrarily far away from the mean of the non-outliers. This is called breakdown; a single outlier may affect the mean arbitrarily strongly. Alternative estimators that don't have this problem and achieve a breakdown point of up to 50% (i.e., you can replace almost 50% of the observations by outliers and still have the estimator within a finite distance to the majority mean) are the median and Huber's M-estimator. Both of these cannot be computed from any sufficient statistic for the Gaussian mean (because the breakdown point argument allows for observations that are not drawn from the same Gaussian to be added - which in reality is very relevant as nothing is perfectly Gaussian). The sufficiency concept relies on the model likelihood and doesn't take any concern into account that could come from allowing anything that doesn't follow the model.

Later addition: @SextusEmpiricus objected that in this case I'm not assuming normality (or in any case the nominal model on which sufficiency is based) anymore, and so the original "sufficient statistic" is no longer sufficient.

I disagree in the sense that the (finite sample) breakdown point as explained above is not a model-based concept, but rather a mathematical characteristic of an estimator. So for example "construct a consistent estimator for the Gaussian mean with a maximum breakdown point" is a mathematically well defined problem involving the Gaussian model (for which the arithmetic mean is sufficient) and no other. The practical relevance of this lies in the possibility that real data may not be precisely Gaussian (in fact they never are), but this doesn't take away from the fact that I technically don't assume any statistical model other than the Gaussian here. The fact that the arithmetic mean is sufficient for the Gaussian distribution is a technical statement, and in my view it makes little sense to say that in order to apply this in practice real data need to be Gaussian (in which case we could never apply it.)

$\endgroup$
5
  • $\begingroup$ @SextusEmpiricus "If that estimator is the median from a sample, then generate a sample with the same information about that parameter and compute it's median." That one isn't the median of the original sample and doesn't have its breakdown properties. $\endgroup$ Commented Jun 17 at 12:29
  • $\begingroup$ Yes, but those properties are only important when there is no sufficient statistic. $\endgroup$ Commented Jun 17 at 12:32
  • $\begingroup$ @SextusEmpiricus Once more, sufficiency is defined in the world of mathematics. It refers to a formal model. The sufficiency definition holds with respect to the assumed model regardless of whether this is true in reality or not. If you insist on interpreting statistical model assumptions in such a way that you are only allowed to apply a model-based method if the model is true, you can never apply any model-based statistics, and any pondering about sufficiency is useless. $\endgroup$ Commented Jun 17 at 12:37
  • 1
    $\begingroup$ In the same way the 'computing other statistics' holds with respect to the assumed model, whether it performs the same in reality or not. $\endgroup$ Commented Jun 17 at 12:49
  • 1
    $\begingroup$ When reasoning off-the-model, there are still assumptions needed for validating some statistical procedures, even the "robust" kind. Whether or (a.s.) not they hold in reality. $\endgroup$
    – Xi'an
    Commented Jun 18 at 8:26
2
$\begingroup$

tl;dr:

It is shown below that a statistical model has no more information for inference of the parameter than the statistical model based on sufficient statistic. That the two models are equivalent, has been shown by using the concept of deficiency, which induces a partial order among a collection of statistical experiments, based on at least as informative as.


Sufficiency Principle:

C&B in $\rm[I]$ explained the Sufficiency Principle as:

If $T(\mathbf X)$ is a sufficient statistic for $\theta,$ then any inference about $\theta$ should depend on the sample $\bf X$ only through the value $T(\mathbf X).$

What follows in this post is a brief discussion of a rigorous framework to comprehend the implication of the above principle.

What is information?

The word “information” is used frequently whenever the topic entails sufficiency. But what does it mean? The following result provides a glimpse:

Result $1.$ Let $(\Omega, \mathfrak A, \mathbb P )$ be a probability space. Consider a random mapping X taking values in $(\mathscr E,\mathfrak E)$. Then a real-valued function $f: \Omega\to\mathbb R$ is $\sigma(X)/\mathfrak B_{\mathbb R}$ measurable if and only if $f= g\circ X,$ where $g$ is a real-valued measurable function on $(\mathscr E,\mathfrak E)$.

If an experimenter is aware about the realization of $X,$ then they also would possess the knowledge of the values of all such $\sigma(X)/\mathfrak B_{\mathbb R}$ measurable functions. In that sense, the information generated by $X$ is associated with $\sigma(X),$ that is, knowing $X$ would only let the experimenter determine those $f$s which are measurable functions of $X.$

Generalizing this to an arbitrary sub-sigma algebra $\mathfrak H\subset \mathfrak A,$ the experimenter has partial information equivalent to $\mathfrak H$ if and only if they are able to tell the values of all measurable functions on $\mathfrak H$ for every $\omega\in \Omega.$

Now while, as noted in $\rm[II, III],$ this seems to be a heuristic interpretation, it is certainly a helpful cosmetic mathematical exposition of an otherwise imprecise usage of the word “information”.

Comparison of Experiments:

Consider a statistical experiment $\mathcal M:= (\mathscr X, \mathfrak A, (P_\theta)_{\theta\in\Theta});$ how much statistical information does this experiment convey?

Unfortunately there is no reasonable way to measure the absolute amount of statistical information corresponding to an experiment. The next feasible approach would then be to develop a criterion (i.e., a partial order) to compare any two given experiments as to how one is “more informative than other” such that the following rational assertion holds: among two experiments $\cal M, ~N $ with same parameter set $\Theta,~\mathcal M$ is at least as informative as $\cal N,$ if corresponding to any decision problem and any decision rule in $\cal N,$ there is a decision rule in $\cal M$ which is at most as risky for any $\theta\in \Theta.$

Definition $1.$ Let $\varepsilon\in\mathbb R_{\geq 0}.$ An experiment $\mathcal M:= (\mathscr X, \mathfrak A, (P_\theta)_{\theta\in\Theta})$ is said to be $\varepsilon$–deficient with respect to $\mathcal N:= (\mathscr Y, \mathfrak B, (Q_\theta)_{\theta\in\Theta})$ if for every subset $F\subseteq \Theta,$ every finite decision space $\mathscr D,$ every loss function $L $ with $\Vert L\Vert_{u}\leq 1$ ($\Vert \cdot\Vert_u$ is the sup norm) and every $\mathsf D_{\mathcal N}:\mathfrak D\times\mathscr Y\to_k [0,1] ,$ there exists a $\mathsf D_{\mathcal M}:\mathfrak D\times\mathscr X\to_k [0,1] $ such that for all $\theta\in F,$ $$\mathsf R(\theta, \mathsf D_{\mathcal M})\leq \mathsf R(\theta, \mathsf D_{\mathcal N}) +\varepsilon.\tag 1\label 1$$

$\eqref 1$ induces a partial order $\succeq^\varepsilon$ in that $\mathcal M \succeq^\varepsilon N;$ if $\varepsilon = 0,~\mathcal M$ is said to be at least as informative as $\mathcal N$ (denoted by $ \succeq$). If $\cal M\succeq N$ and $\cal N\succeq M,$ the statistical experiments are equivalent and denoted by $\cal M\sim N.$

The concept of derived/subexperiment is of necessity now. This follows from randomization of a statistical experiment.

Definition $2.$ Let $(\mathscr Y,\mathfrak B)$ be a measurable space and let $\mathsf K:\mathfrak B\times \mathscr X\to_k [0,1]$ be a Markov kernel. Then $\mathcal N\equiv \mathsf K\mathcal M:= (\mathscr Y,\mathfrak B,(\mathsf KP_\theta)_{\theta\in \Theta})$ is the randomization of the statistical experiment $\mathcal M.$

Lemma $1.$ Let $\mathcal M:= (\mathscr X, \mathfrak A, (P_\theta)_{\theta\in\Theta})$ and $\mathcal M:= (\mathscr Y, \mathfrak B, (Q_\theta)_{\theta\in\Theta})$ be two statistical experiments. If there is a Markov kernel $\mathsf K:\mathfrak B\times \mathscr X\to_k [0,1]$ such that for each $\theta\in\Theta, ~\Vert \mathsf KP_\theta-Q_\theta\Vert\leq \varepsilon,$ then $\mathcal M \succeq^\varepsilon N;$ if $\mathcal N = \mathsf K\mathcal M,$ then $\mathcal M\succeq N.$

Prelude to Sufficiency:

Consider $\mathsf K:= \delta_{T(x)},$ where $T:\mathscr X\to_m \mathscr Y.$ Does there exist any decision rule $\mathsf C: \mathfrak D\times \mathscr Y\to_k [0,1]$ for the reduced experiment $\mathcal G:= \mathsf K\mathcal M = (\mathscr Y, \mathfrak B, (Q_\theta := P_\theta\circ T^{-1})_{\theta\in\Theta})$ that factorizes $\mathsf D$ viz. $\mathsf D(\cdot , x) = \mathsf C(\cdot, T(x))$ (the implication of which is that this decision rule would depend on the observations only through the value of $T,$ in league of the Sufficiency Principle)? Following delivers a sufficient condition for that.

Theorem $1.$ Let $\mathcal M$ be dominated by $Q,$ say and let the decision space be Borel space. If the densities $\frac{\mathrm dP_\theta}{\mathrm dQ}$ are measurable with respect to $\sigma(T),$ then for every decision rule $\mathsf D,$ there exists a factorized decision rule, say, $\mathsf C\circ T$ such that $\mathsf R(\theta, \mathsf D) = \mathsf R(\theta, \mathsf C\circ T).$ In particular $\cal G\sim M.$

Flavors of Sufficiency:

Definition $3.1.$ Let $\mathcal M $ be a statistical experiment and $(\mathscr Y,\mathfrak B)$ be a measurable space. A statistic $ T:\mathscr X\to_m \mathscr Y $ is sufficient for $(P_\theta)_{\theta\in\Theta}$ if for every $A\in\mathfrak A,$ there exists a function $\kappa_A:\mathscr Y\to\mathbb R$ such that for each $\theta\in\Theta$ almost surely $P_\theta$ $$\mathbf E_\theta(1\!\!1_A\mid T) = \kappa_A(T).$$

Definition $3.2.$ $T$ is regular sufficient for $(P_\theta)_{\theta\in\Theta}$ if there exists a Markov kernel $\mathsf M: \mathfrak A\times \mathscr Y\to_k [0,1]$ such that for each $\theta\in\Theta$ almost surely $P_\theta$ $$\mathbf E_\theta(1\!\!1_A\mid T) = \mathsf M(A, T).$$

Regular sufficiency implies there exists a regular conditional distribution of $A\in\mathfrak A$ given $T$ independent of $\theta: $ $$P_\theta(A\cap T^{-1}(B)) = \int_{\mathscr Y} {1\!\!1}_{B}(t)\mathsf M(A, t)~P_\theta\circ T^{-1}(\mathrm dy), ~B\in\mathfrak B .\tag 2\label 2$$

It is easy to see $\eqref 2$ further implies $\mathcal L(X, T(X)) = \mathsf M\otimes (P_\theta\circ T^{-1}). $ This means the marginal distribution $P_\theta=\mathsf M (P_\theta\circ T^{-1}). $

Definition $3.3$ $T$ is Blackwell sufficient for $(P_\theta)_{\theta\in\Theta}$ if there exists a Markov kernel $\mathsf M: \mathfrak A\times \mathscr Y\to_k [0,1]$ such that for all $\theta\in\Theta, P_\theta=\mathsf M (P_\theta\circ T^{-1}). $

Thus, regular sufficiency implies Blackwell sufficiency but not vice-versa even for sufficiency, in general. However,

Theorem $2.$ Let $(\mathscr X, \mathfrak A)$ be a Borel space and $\mathcal M$ be dominated by a sigma-finite measure. Then sufficiency, regular sufficiency, Blackwell sufficiency are equivalent.

By Lemma $1. ~\mathcal M\succeq \mathcal G.$ If $T$ is Blackwell sufficient, then there exists a Markov kernel such that $P_\theta= \mathsf MQ_\theta $ which means $\mathcal M\sim \mathcal G.$ In fact, for $T$ being Blackwell sufficient, Theorem $1.$ becomes necessary condition too for two experiments for being mutual randomization of each other and hence being equivalent.

Finally what about the discussion involving an experimenter generating a “fake” data based on the sufficient statistic, which has the same unconditional probability distribution as that of the original sample and thus being a replica of the latter?

Theorem $3.$ Let $(\mathscr Y,\mathfrak B), ~(\mathscr Z,\mathfrak C )$ be a measurable space and Borel space respectively and let $\mathsf M:\mathfrak C\times \mathscr Y\to_k[0,1]$ be a Markov kernel. Then there exists a measurable function $m: \mathscr Y\times (0,1)\to \mathscr Z$ such that for each $y\in\mathscr Y,$ the probability measure $\mathsf M(\cdot, y)$ is induced by the uniform measure $\mathrm U$ and $\varrho $ defined as $u\mapsto m(y,u) ,$ that is, for $C\in\mathfrak C,$ $$\mathsf M(C, y) = \mathrm U\circ \varrho^{-1}(C).\tag 3\label 3$$

In the current context when $T$ is Blackwell sufficient, the above theorem implies $(P_\theta\otimes \mathrm U)\circ((x,u)\mapsto m(T(x), u))^{-1} = P_\theta,$ that is, with the knowledge of $T(x)$ and a realization of a uniform variable, $m$ has the same distribution as that of the original sample.

Appendix:

Remarks:

  • Let the underlying sample space be $(\Omega, \mathfrak F, (\mathbb P_\theta)_{\theta\in\Theta}). $ For an experiment $\cal M,$ a decision problem involves a decision space $(\mathscr D, \mathfrak D)$ and a decision rule $\mathsf D$ which is a Markov kernel $\mathfrak D\times \mathscr X\mapsto_k [0,1]. $ A loss function $L$ is a real-valued function $\Theta\times \mathscr D\to \mathbb R$ such that for each $\theta\in\Theta,~L(\theta, \cdot)$ is $\mathfrak D/\mathfrak B_{\mathbb R}$ measurable. When the data $X:\Omega\to_m \mathscr X$ is realized, the experimenter takes action $A:\Omega\to_m\mathscr D,$ the value of which depends on the realization of $X$ and is governed by $\mathsf D(\cdot, x),$ that is, $\mathcal L((A, X)\mid \mathbb P_\theta)= \mathsf D\otimes P_\theta.$ The risk function $\mathsf R$ of a decision rule $\sf D$ for $\theta\in\Theta$ is $\mathsf R(\theta, \mathsf D):=\mathbf E_\theta L(\theta, A) = \int\left[\int L(\theta, a)\mathsf D(\mathrm da, x)\right]~P_\theta(\mathrm dx).$
  • A Blackwell Sufficient statistic is also sometimes called Exhaustive statistic.

Proofs:

  • Lemma $1.$ Observe for $\theta\in\Theta,$ \begin{align} \mathsf R(\theta, \mathsf D_{\mathcal N}) &= \int_{\mathscr Y}\left[\int_{\mathscr D} L(\theta, a)\mathsf D_{\mathcal N}(\mathrm da, y)\right]~(\mathsf KP_\theta)(\mathrm dy)\\ &= \int_{\mathscr X}\left[\int_{\mathscr Y} \left[\int_{\mathscr D}L(\theta, a) \mathsf D_{\mathcal N}(\mathrm da, y )\right]\mathsf K(\mathrm dy, x)\right]P_\theta(\mathrm dx)\\ &= \mathsf R(\theta, (\mathsf D_{\mathcal N}\mathsf K)_{\mathcal M}).\end{align} The lemma follows by noting that $\mathsf R(\theta, (\mathsf D_{\mathcal N}\mathsf K)_{\mathcal M})\leq \mathsf R(\theta, \mathsf D_{\mathcal N}) + \Vert \mathsf KP_\theta-Q_\theta\Vert$ for $\Vert L\Vert_u \leq 1.$

  • Theorem $1.$ As $(\mathscr D,\mathfrak D)$ is a Borel space, there exists a Markov kernel $\mathsf C: \mathfrak D\times \mathscr Y\to_k [0,1]$ such that $$\mathcal L((A, X)\mid \mathbb P) = \mathsf C\otimes (Q\circ T^{-1}) .$$ By assumption for some $g_\theta:\mathscr Y\to_m \mathbb R, ~\frac{\mathrm dP_\theta}{\mathrm dQ} = g_\theta(T(x)).$ For $h: \mathscr D\times \mathscr Y\to \mathbb R_{\geq 0},$
    \begin{align}\mathbf Eh(A, T(X)) &= \int_{\mathscr X}\left[\int_{\mathscr D}h(a, T(x))\mathsf D(\mathrm da, x)\right]~Q(\mathrm dx)\\ &= \int_{\mathscr X}\left[\int_{\mathscr D}h(a, T(x))\mathsf C(\mathrm da, T(x))\right]~Q(\mathrm dx);\end{align} taking $h\equiv Lg_\theta,$ \begin{align}\mathsf R(\theta, \mathsf C\circ T) &= \int_{\mathscr X} \left[\int_{\mathscr D} L(\theta, a)\mathsf C(\mathrm da, T(x))\right]~g_\theta(x)Q(\mathrm dx)\\ &= \int_{\mathscr X} \left[\int_{\mathscr D} L(\theta, a)\mathsf D(\mathrm da, x)\right]~g_\theta(x)Q(\mathrm dx)\\ &= \mathsf R(\theta, \mathsf D).\end{align} By Lemma $1.,~\mathcal M\succeq \mathcal G$ and since the decision space is Borel and finite, the first part ensures $\mathcal G\succeq \mathcal M.$

  • Theorem $2.$ It is sufficient to show sufficiency implies regular sufficiency. It follows from the more general result: If $T: (\mathscr X,\mathfrak A)\to_m(\mathscr Y,\mathfrak B)$ is sufficient, then for every statistic $S: (\mathscr X,\mathfrak A)\to_m (\mathscr Z,\mathfrak C)$ where $(\mathscr Z,\mathfrak C)$ is a Borel space, then there exists a regular conditional distribution , given $T,$ which is independent of $P_\theta.$ This is a straightforward application of factorization theorem and this CV.SE answer.

  • Theorem $3.$ Let $\varphi:\mathscr Z\to_m(0,1) $ and let $\psi: (0,1)\to_m \mathscr Z $ such that $\psi(\varphi( z)) = z$ for $z\in\mathscr Z.$ Define for any measure $\mu$ on $\mathfrak C $ another measure $\nu := \mu\circ \varphi^{-1}$ on $\mathfrak B_{(0,1)}.$ It is easy to see then $\mu = \nu\circ \psi^{-1}.$

    Consider a Markov kernel $\mathsf N: \mathfrak B_{(0,1)}\times \mathscr Y\to_k[0,1];$ define $F(y,t):= \mathsf N((0,t], y)$ for $y\in\mathscr Y, ~t\in (0,1).$ For each $t\in(0,1), ~y\mapsto F(y,t)$ is measurable and for $y\in \mathscr Y, ~t\mapsto F(y,t)$ is non-decreasing and right continuous. Define $G(y, u) := \inf\{t\in(0,1)\mid F(y,t)\geq u\}; ~G(y,u)\leq t\iff F(y,t)\geq u. ~G$ is $\mathfrak B\times\mathfrak B_{(0,1)}$– measurable and $\mathrm U\{u\in(0,1)\mid G(y,u)\leq t\} = \mathrm U\{u\in(0,1)\mid F(y,t)\geq u\} = F(y,t)$. This means $\mathsf N(B,y) = \mathrm U\{u\in (0,1)\mid G(y,u)\in B\}, ~B\in\mathfrak B_{(0,1)}.$ The proof concludes by taking $\mathsf N(B, y) = \mathsf M(\varphi^{-1}(B), y)$ and $m(y, u) = \psi(G(y,u)).$

--

References:

$\rm[I]$ Statistical Inference, George Casella, Roger L. Berger, Wadsworth, $2002,$ sec. $6.2.$

$\rm[II]$ Probability and Stochastics, Erhan Çinlar, Springer Science$+$Business, $2011,$ sec. $\rm II.4.$

$\rm[III]$ Probability and Measure, Patrick Billingsley, John Wiley & Sons, $2012,$ sec. $34.$

$\rm[IV]$ Comparison of Statistical Experiments, Erik Torgersen, CUP, $1991,$ sec. $6.1.$

$\rm[V]$ Statistical Decision Theory: Estimation, Testing and Selection, Friedrich Liese, Klaus-J. Miescke, Springer Science$+$Business, chap. $4.$

$\rm[VI]$ Parametric Statistical Theory, Johann Pfanzagl, Walter de Gruyter & Co., $1994,$ sec. $1.1-1.3, ~1.10.$

$\endgroup$
21
  • 1
    $\begingroup$ This might elucidate the question from the point of view of a graduate student in statistics - I have no doubt it would. But it would not help anyone lacking a measure-theoretic understanding of probability. $\endgroup$ Commented Jun 20 at 22:41
  • 1
    $\begingroup$ I would add a tl;dr to it to say what is happening. The point was that you will get many discussions here, and at other places mentioning about "information", and then generating another data based on the sufficient statistic, having the same distribution as the original sample, but it is hard to come across a discussion where there is an actual mathematical framework that complements this whole story. My idea was to present a brief post in that vein and in fact, I ensured that the measure-theoretic machinary is at its elementary form here. @AdrianKeister $\endgroup$ Commented Jun 21 at 3:36
  • 2
    $\begingroup$ I think its great to have mathematical answers like this alongside answers that require less mathematical expertise. It can also be frustrating, I know from experience, when people keeping using hand-wavy terms when it is the very handy-wavy-ness that is impeding my learning. $\endgroup$
    – Galen
    Commented Jun 21 at 4:24
  • 2
    $\begingroup$ @Sextus (with our perennial disagreement with level of rigor), my main intention was to provide a mathematical framework to your answer (the part of factorization theorem) and the one by ogustavo (where they constructed a "fake" data, that is replica of the original based on the sufficient statistic). I have also have to keep in mind the post is self-contained. It has been written chronologically and neither part of it is irrelevant here, imo. As for the operational definition of sufficiency that you mentioned, it follows from def. $3.1.$ $\endgroup$ Commented Jun 21 at 10:52
  • 1
    $\begingroup$ Meanwhile, there are enough materials in $\rm [IV], [V], [VI]$ for you to enjoy @rfloc. In fact, going through, you could get some other references too. $\endgroup$ Commented Jun 26 at 20:57
1
$\begingroup$

Let me give another perspective that may help. This is also qualitative, but there is a rigorous version of that particularly important in Information Theory - known as Markov property.

In the beginning, we have two objects, data (coming from a Random Variable, call it X) and parameter, $\theta$ (another rv, implicitly assumed since we are talking about its estimator). These two, are assumed to be dependent (otherwise, there is no point in trying to estimate one from the other). Now, the third object enters the game, Sufficient Statistic, T. The intuitive idea when we say T is enough to estimate $\theta$ really means that if we know T (ie conditioned on T), X provides no additional info, that is, X and $\theta$ are independent. In other word, knowledge of X is equivalent to knowledge of T as far as estimation of $\theta$ is concerned. Note that in probabilities are where all the uncertainties are captured, and hence "any estimate" when (conditional) probabilities are independent (eg conditional densities factorize).

$\endgroup$
1
$\begingroup$

The second sentence in the quote is proven by the factorization theorem--which shows that a sample conditioned on its sufficient statistic is independent of the parameter. The first sentence is equivalent to the second sentence (to the extent that an informal statement can be equivalent to a mathematical statement) if we modify it to say "all the parameter information." In other words, the statement assumes that only parameter (Fisher) information is inferentially useful. Or, at least, it assumes any other type of information, however conceived and defined and measured, must be redundant with the Fisher information for purposes of improving the accuracy and/or precision of an estimate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.