# Sufficient statistic definition in Koller's Probabilistic Graphical Models

In Daphne Koller's Probabilistic Graphical Models, the sufficient statistic is defined as follows (p 721):

A function $$\tau(\xi)$$ from instances of $$\chi$$ to $$\mathbb R^l$$ (for some $$l$$) is a sufficient statistic if, for any two data sets $$\mathcal D$$ and $$\mathcal D'$$ and any $$\theta \in \Theta$$, we have that: $$\sum_{\xi[m] \in \mathcal D} \tau(\xi[m]) = \sum_{\xi[m] \in \mathcal D'} \tau(\xi[m]) \implies L(\theta: \mathcal D) = L(\theta: \mathcal D')$$

We often refer to the tuple $$\sum_{\xi[m] \in \mathcal D} \tau(\xi[m])$$ as the sufficient statistics of the data set $$\mathcal D$$.

Later, it gives an example that for the Gaussian distribution, $$\tau(x)=<1, x, x^2>$$ and so for sample $$x_1, \dots, x_M$$, the sufficient statistic for the data set is$$\sum_m \tau(x_m)=$$

The definition of sufficient statistic that I find in other sources seems different. For example, in Casella and Berger's Statistical Inference, it is defined as follows:

A statistic $$T(X)$$ is a sufficient statistic for $$\theta$$ if the conditional distribution of the sample $$X$$ given the value of $$T(X)$$ does not depend on $$\theta$$.

Note: the sample $$X$$ in the above is a vector consisting of $$n$$ data points drawn from some distribution.

I have three specific questions about the definition in the Probabilistic Graphical Model:

1. As far as I know, a sample of $$m$$ data points is a vector $$(\xi[1], \xi[2], \dots, \xi[m])$$, and a statistic on a sample is a function that maps this vector to $$\mathbb R^l$$. A sufficient statistic is a statistic, and so should be a mapping from $$(\xi[1], \xi[2], \dots, \xi[M])$$ to $$\mathbb R^l$$, like in the second definition. But in the PGM definition, it is a mapping $$\tau$$ from a single data point $$\xi[m]$$ to $$\mathbb R^l$$.

2. It later defined $$\sum_{\xi[m] \in \mathcal D} \tau(\xi[m])$$ as the sufficient statistics (plural) of the data set $$\mathcal D$$. This certainly is equivalent to the $$T(X)$$ in the second definition. But I am not sure the difference between a sufficient statistic (for the model) and sufficient statistics of data set $$\mathcal D$$.

3. The definition seems to be referring to a general case that is true for all distributions. Then I am not sure in $$\sum_{\xi[m] \in \mathcal D} \tau(\xi[m])$$, why $$\tau(\xi[m])$$ for each data point can be summed. I feel this can only be true for the exponential family.

• To digress a bit, as you pointed out, for two datasets $D$ and $D'$ with the same sufficient static, the likelihoods do not have to be equal. But for all the examples that I can come up with, $D$ and $D'$ have different numbers of data points. Is there an example where the likelihoods are different but $D$ and $D'$ have the same number of data points? Jan 5, 2021 at 8:04
• A normal sample with unknown mean would do: if $\bar{X}(D)=\bar{X}(D^\prime)$, the sum of squares could differ between $D$ and $D^\prime$. Jan 5, 2021 at 14:01
• Ah, that makes a lot of sense. Jan 5, 2021 at 14:13
• djsaunde.github.io/read/books/pdfs/… Jan 5, 2021 at 22:57

1. A sufficient statistic for a sample of size $$M$$ is not always the sum of statistics of the elements of the sample. A counter-example is the Uniform $$\mathcal U(0,\theta)$$ distribution where a sufficient statistic is $$\max(\xi_1,\ldots,\xi_M)$$
2. When two samples of the same size return the same value of a sufficient statistic, their respective likelihood functions are proportional and not equal. This is a consequence of the factorisation theorem and is exemplified by the Normal $$\mathcal N(\theta,1)$$ model.
3. This definition excludes non-linear bijective transforms of sufficient statistics as being sufficient. Even for exponential families, it forces the sufficient statistic to be based on the natural statistics, i.e., $$\tau:\mathfrak X\mapsto\mathbb R^\ell$$ such that the density of the sample points writes as $$p(x|\theta)\propto \exp\{\theta^\text{T}\tau(x)\}$$ [wrt to a fixed measure].
4. Although this is not incorrect, the sample size $$M$$ has no place in a sufficient statistic as it is purely ancillary and non-random in the examples covered in that section.
As for the statistic-versus-statistics (minor) incoherence, it sounds to me that a statistic is first defined as a vector in $$\mathbb R^\ell$$ and then the "tuple" [which should be an $$\ell$$-uple!] is a vector made of components that are also statistics, hence a "tuple of sufficient statistics" [which would read better as a sufficient tuple of statistics!]