4
$\begingroup$

Migrated from MSE

I am trying to understand the following intuition for sufficient statistics in Casella & Berger (2nd edition, pg. 272):

A sufficient statistic captures all of the information about $\theta$ in this sense. Consider Experimenter 1, who observes $\mathbf X=\mathbf x$ and, of course, can compute $T(\mathbf X)=T(\mathbf x)$. To make an inference about $\theta$ he can use the information that $\mathbf X=\mathbf x$ and $T(\mathbf X)=T(\mathbf x)$. Now consider Experimenter 2, who is not told the value of $\mathbf X$ but only that $T(\mathbf X)=T(\mathbf x)$. Experimenter 2 knows $P(\mathbf X=\mathbf y|T(\mathbf X)=T(\mathbf x))$, a probability distribution on $A_{T(\mathbf x)}=\{\mathbf y:T(\mathbf y)=T(\mathbf x)\}$, because this can be computed from the model without knowledge of the true value of $\theta$. Thus, Experimenter 2 can use this distribution and a randomization device, such as a random number table, to generate an observation $\mathbf Y$ satisfying $P(\mathbf Y=\mathbf y|T(\mathbf X)=T(\mathbf x))=P(\mathbf X=\mathbf y|T(\mathbf X)=T(\mathbf x))$. It turns out that, for each value of $\theta$, $\mathbf X$ and $\mathbf Y$ have the same unconditional probability distribution, as we shall see below. So Experimenter 1, who knows $\mathbf X$, and Experimenter 2, who knows $\mathbf Y$, have equivalent information about $\theta$.

I understand that Experimenter 1 has access to the original observation $\mathbf x$ and can compute the sufficient statistic $T(\mathbf x)$, while Experimenter 2 is given only $T(\mathbf x)$. Furthermore, I understand that by the definition of sufficient statistics the conditional distribution $P(\mathbf X=\mathbf y|T(\mathbf X)=T(\mathbf x))$ is independent of $\theta$ and thus only knowledge of $T(\mathbf x)$ is needed to generate random samples from this distribution. Where I get hung up is why the fact that $\mathbf X$ and $\mathbf Y$ having the same unconditional distribution proves both experimenters have the same information about $\theta$. I thought $\mathbf Y$ was distributed according to the conditional distribution and not the unconditional distribution. A proof of the fact that both $\mathbf X$ and $\mathbf Y$ have the same unconditional distribution is given further down in the text and, while I think I understand the proof, it doesn't feel like I have developed and understand the intuition here.

Perhaps an alternative explanation in someone else's words will get me over the hurdle here.

$\endgroup$
4
  • 2
    $\begingroup$ Hm. There are so many posts on sufficient statistic here. Have you checked any? $\endgroup$ Commented Jul 11 at 12:10
  • 1
    $\begingroup$ The sentence "$𝐘$ was distributed according to the conditional distribution and not the unconditional distribution" is not truly making sense within probability theory: $𝐘$ is both distributed from the unconditional distribution $P_\theta(𝐘)$ and from the conditional distribution $𝑃(𝐗=𝐲|𝑇(𝐗)=𝑇(𝐱))$ given $T(𝐗)=𝑇(𝐱)$ $\endgroup$
    – Xi'an
    Commented Jul 11 at 15:28
  • 1
    $\begingroup$ "while I think I understand the proof, it doesn't feel like I have developed and understand the intuition here." The intuition is part of the duplicate question and several answers there tackle the idea of 'the distribution of the data $X$' and 'an alternative set of simulated data $Y$ generated by only being based on the (observed) sufficient statistic'. $\endgroup$ Commented Jul 11 at 18:05
  • 1
    $\begingroup$ "Where I get hung up is why the fact that X and Y having the same unconditional distribution proves both experimenters have the same information about θ" This is not accurate. X and Y do not provide the same information, because they have the same unconditional distribution. For example two seperate measurements X1 and X2 have the same distribution, but may give different information. It also depends on how you define information, but probably more precise would be to say that T(X) and X provide the same information. $\endgroup$ Commented Jul 11 at 18:24

2 Answers 2

2
$\begingroup$

One can visualize this explanation along a temporal axis

  1. The random variable $\mathbf X$ is generated by a distribution $\mathbb P_\theta(\mathbf x)$ with unknown parameter $\theta$ and its observed realization is $\mathbf x$
  2. The value of a sufficient statistic $T(\cdot)$, namely $T(\mathbf x)$, is deduced
  3. A new random variable $\mathbf Y$ is generated from the conditional distribution $\mathbb P_\theta(\mathbf y|T(\mathbf Y)=T(\mathbf x))$, which does not depend on $\theta$ by the very definition of sufficiency, with realization $\mathbf y$

Then

  1. $\mathbf Y$ and $\mathbf X$ share the same value for the sufficient statistic, $T(\mathbf y)=T(\mathbf x)$, by construction
  2. $\mathbf Y$ and $\mathbf X$ share the same marginal distribution $P_\theta(\cdot)$, by symmetry
  3. the information on $\theta$ contained in either the observed $\mathbf Y$ or the observed $\mathbf X$ is the same, again by symmetry

For instance, consider $\mathbf X=(X_1,X_2)$ made of iid Gaussians $\mathcal N(\mu,1)$. Then $\bar X=(X_1+X_2)/2$ is sufficient for $\mu$, with realization $\bar x$. One can then generate $\mathbf Y=(Y_1,Y_2)$ sharing the same value of the sufficient statistic as $\mathbf x$, by simulating $$Y_1\sim\mathcal N(\bar x,1/2)$$ as explained in that entry. And then setting $Y_2=2\bar x-Y_1$. By construction, the realisations of both $\mathbf X$ and $\mathbf Y$ share the same value of the sufficient statistic.

$\endgroup$
2
  • 1
    $\begingroup$ Xi'an, last day I added the new edition link of C&B in your post. Perhaps you have edited it in the recent edit. It's not working unfortunately. Could you have a look? $\endgroup$ Commented Jul 12 at 13:18
  • 1
    $\begingroup$ I noticed 6 from the url inadvertently got deleted in your last edit. I have fixed that. $\endgroup$ Commented Jul 12 at 13:23
1
$\begingroup$

What is a sufficient statistic? What is its objective?

To understand that, consider a (Polish) measurable space $(\mathscr X,\mathfrak A ) $ let $\mathfrak P$ be a collection of probability measures on $\mathfrak A. $ In general, a statistical procedure based on a statistic $T:(\mathscr X, \mathfrak A) \to_m (\mathscr Y, \mathfrak B) $ that is, $T(x) $ plausibly shouldn't be equivalent to one based on $x, $ owing to the former losing information as $x\mapsto T(x). $

If $T$ is able to generate, via randomization, a new random variable $\hat x,$ based on which, for every $P\in\mathfrak P, $ the statistical procedure would be equivalent to the ones based on $x, $ then it would be sufficient to work with $T$ only and naturally then $\hat x$ would have the same distribution as that of $x.$

Formally, $T$ is a sufficient statistic for $\frak P$ if there exists a Markov kernel $\mathsf M\mid \mathscr Y\times \mathfrak A$ for each $P\in\mathfrak P$ such that $$\int \mathsf M(y, A) ~P\circ T^{-1}(\mathrm dy) =P(A),~A\in\mathfrak A. $$

Now by Theorem $3.$ proved here, there exists a function $m:\mathscr Y\times(0, 1) \to\mathscr X$ such that for each $y\in\mathscr Y, $ the induced distribution of $u\mapsto m(y, u) $ under the uniform measure $\mathrm U$ is identical to $\mathsf M(y, \cdot). $ So, for a sufficient statistic $T, $ $$(P\otimes \mathrm U) \circ((x,u)\mid m(T(x),u)\in A)=P(A),~~P\in\mathfrak P. $$

What does this mean?

It implies once one knows $T(x) $ and there is a realization of an auxiliary uniform random variable on $(0, 1) , $ then one can obtain with $m(T(x), u) $ a random variable that has exactly the same distribution as the original $x$ for every $P\in\mathfrak P, $ which is agreement as was conceived earlier in the motivation.

Speaking simply, in context of C&B, one can construct $\mathbf Y$ equivalent to $\mathbf X$ based on the knowledge of $T$ alone for randomization doesn't add any new information about $\theta.$

--

Reference:

Parametric Statistical Theory, Johann Pfanzagl, Walter de Gruyter & Co., $1994,$ sec. $1.1.$

$\endgroup$
4
  • 4
    $\begingroup$ I am not sure if this exposition is the best way to further intuition beyond what the OP has already taken from CB. $\endgroup$ Commented Jul 11 at 14:05
  • $\begingroup$ I have added the motivation, the intuition associated with the concept of sufficiency as a whole. Even if OP wants to leave the formal structure, the exposition associated with it has been dealt with here. $\endgroup$ Commented Jul 11 at 14:13
  • 1
    $\begingroup$ I am not well versed in measure theory and, while this answer looks intriguing, I cannot understand it well. $\endgroup$ Commented Jul 11 at 16:15
  • $\begingroup$ @AaronHendrickson, I have added what I felt the necessary soul of the argument presented in C&B. The point is once you know about $T$ you cannot get any new information about $\theta$ other than what $T$ provides. As mentioned in the answer, you can then construct a new random variable based on randomization which is equivalent to the original sample. The information of $\theta$ has been provided by $T.$ The equivalence necessitates their unconditional distribution being same. $\endgroup$ Commented Jul 11 at 17:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.