In Daphne Koller's Probabilistic Graphical Models, the sufficient statistic is defined as follows (p 721):

A function $\tau(\xi)$ from instances of $\chi$ to $\mathbb R^l$ (for some $l$) is a sufficient statistic if, for any two data sets $\mathcal D$ and $\mathcal D'$ and any $\theta \in \Theta$, we have that: $$ \sum_{\xi[m] \in \mathcal D} \tau(\xi[m]) = \sum_{\xi[m] \in \mathcal D'} \tau(\xi[m]) \implies L(\theta: \mathcal D) = L(\theta: \mathcal D') $$

We often refer to the tuple $\sum_{\xi[m] \in \mathcal D} \tau(\xi[m])$ as the sufficient statistics of the data set $\mathcal D$.

Later, it gives an example that for the Gaussian distribution, $\tau(x)=<1, x, x^2>$ and so for sample $x_1, \dots, x_M$, the sufficient statistic for the data set is$$\sum_m \tau(x_m)=<M, \sum_m x, \sum_m x^2 > $$

The definition of sufficient statistic that I find in other sources seems different. For example, in Casella and Berger's Statistical Inference, it is defined as follows:

A statistic $T(X)$ is a sufficient statistic for $\theta$ if the conditional distribution of the sample $X$ given the value of $T(X)$ does not depend on $\theta$.

Note: the sample $X$ in the above is a vector consisting of $n$ data points drawn from some distribution.

I have three specific questions about the definition in the Probabilistic Graphical Model:

  1. As far as I know, a sample of $m$ data points is a vector $(\xi[1], \xi[2], \dots, \xi[m])$, and a statistic on a sample is a function that maps this vector to $\mathbb R^l$. A sufficient statistic is a statistic, and so should be a mapping from $(\xi[1], \xi[2], \dots, \xi[M])$ to $\mathbb R^l$, like in the second definition. But in the PGM definition, it is a mapping $\tau$ from a single data point $\xi[m]$ to $\mathbb R^l$.

  2. It later defined $\sum_{\xi[m] \in \mathcal D} \tau(\xi[m])$ as the sufficient statistics (plural) of the data set $\mathcal D$. This certainly is equivalent to the $T(X)$ in the second definition. But I am not sure the difference between a sufficient statistic (for the model) and sufficient statistics of data set $\mathcal D$.

  3. The definition seems to be referring to a general case that is true for all distributions. Then I am not sure in $\sum_{\xi[m] \in \mathcal D} \tau(\xi[m])$, why $\tau(\xi[m])$ for each data point can be summed. I feel this can only be true for the exponential family.

I would appreciate some clarifications about this concept.

  • $\begingroup$ To digress a bit, as you pointed out, for two datasets $D$ and $D'$ with the same sufficient static, the likelihoods do not have to be equal. But for all the examples that I can come up with, $D$ and $D'$ have different numbers of data points. Is there an example where the likelihoods are different but $D$ and $D'$ have the same number of data points? $\endgroup$ Jan 5, 2021 at 8:04
  • $\begingroup$ A normal sample with unknown mean would do: if $\bar{X}(D)=\bar{X}(D^\prime)$, the sum of squares could differ between $D$ and $D^\prime$. $\endgroup$
    – Xi'an
    Jan 5, 2021 at 14:01
  • $\begingroup$ Ah, that makes a lot of sense. $\endgroup$ Jan 5, 2021 at 14:13
  • 1
    $\begingroup$ djsaunde.github.io/read/books/pdfs/… $\endgroup$ Jan 5, 2021 at 22:57

1 Answer 1


This definition of sufficiency by Koller & Friedman is both restrictive and incorrect:

  1. A sufficient statistic for a sample of size $M$ is not always the sum of statistics of the elements of the sample. A counter-example is the Uniform $\mathcal U(0,\theta)$ distribution where a sufficient statistic is $\max(\xi_1,\ldots,\xi_M)$
  2. When two samples of the same size return the same value of a sufficient statistic, their respective likelihood functions are proportional and not equal. This is a consequence of the factorisation theorem and is exemplified by the Normal $\mathcal N(\theta,1)$ model.
  3. This definition excludes non-linear bijective transforms of sufficient statistics as being sufficient. Even for exponential families, it forces the sufficient statistic to be based on the natural statistics, i.e., $\tau:\mathfrak X\mapsto\mathbb R^\ell$ such that the density of the sample points writes as $p(x|\theta)\propto \exp\{\theta^\text{T}\tau(x)\}$ [wrt to a fixed measure].
  4. Although this is not incorrect, the sample size $M$ has no place in a sufficient statistic as it is purely ancillary and non-random in the examples covered in that section.

As for the statistic-versus-statistics (minor) incoherence, it sounds to me that a statistic is first defined as a vector in $\mathbb R^\ell$ and then the "tuple" [which should be an $\ell$-uple!] is a vector made of components that are also statistics, hence a "tuple of sufficient statistics" [which would read better as a sufficient tuple of statistics!]

  • 1
    $\begingroup$ I think the book is trying to define sufficient statistic per data point, which is something not attainable in the general case. $\endgroup$ Jan 6, 2021 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.