I'm reading about sufficient statistics, and have come across two definitions which seems unrelated, and I'm trying to understand their connection.

The first definition is from Wikipedia

A statistic $t = T(X)$ is sufficient for underlying parameter $\theta$ precisely if the conditional probability distribution of the data $X$, given the statistic $t = T(X)$, does not depend on the parameter $\theta$ $$ \operatorname{Pr}(x \mid t, \theta) = \operatorname{Pr}(x \mid t) $$

Wikipedia also provides the following equivalent statements, which I assume is due to bayes theorem $$ \operatorname{Pr}(\theta \mid t, x) = \operatorname{Pr}(\theta \mid t) \\ \operatorname{Pr}(\theta, x \mid t) = \operatorname{Pr}(\theta \mid t)\operatorname{Pr}(x \mid t) $$

The other is an informal description of the theorem, also from Wikipedia

Roughly, given a set $\mathbf {X}$ of independent identically distributed data conditioned on an unknown parameter $\theta$, a sufficient statistic is a function $T(\mathbf{X})$ whose value contains all the information needed to compute any estimate of the parameter (e.g. a maximum likelihood estimate).

which I would understand as $\mathcal{I}(\theta \mid t,x) = \mathcal{I}(\theta \mid t)$ for some information measure, i.e. that the information of $\theta$ given $t$ is the same as if we were given $t$ and $x$, so $t$ is sufficient.

The page on Fisher Information (Wikipedia) shows that the same Fisher information can be inferred from $X$ and $T(X)$, but Fisher information is not mentioned on the page on sufficient statistics. It is not clear if this is the only formal definition of information, or if this holds for other kinds of information.

As there is already a question on the intuition behind the equivalence of the statements, an answer to this question should contain the answer to:

  • How is information defined in this context? Is it Fisher information or something more general?
  • How may one demonstrate using the definition of information and the definition of sufficient statistic, that the two statements are equivalent? (or not?)

A proof of equivalence should preferably be as rigorous as possible, but not incomprehensible.

  • 1
    $\begingroup$ Why the downvote? If there's anything I can do to improve the question, I'd like to know. $\endgroup$
    – Frank Vel
    Aug 29, 2018 at 21:10
  • $\begingroup$ The duplicate question does not seem to have an accepted answer. The only answers are based on intuition and are not rigorous, which would be preferable. $\endgroup$
    – Frank Vel
    Aug 30, 2018 at 9:55
  • 2
    $\begingroup$ Because Wikipedia does not use "all the information needed" in any rigorous or mathematically defined sense that goes beyond the answers in the duplicate, it's fruitless to look for anything more. It's immaterial whether the duplicate has an accepted answer. Indeed, the highest voted answer was posted by the original proposer! $\endgroup$
    – whuber
    Aug 30, 2018 at 19:15
  • 1
    $\begingroup$ I'm confused. @FrankVel, did you flag your own question as a duplicate? And then argue against it being a duplicate? $\endgroup$
    – mkt
    Aug 31, 2018 at 7:42
  • 1
    $\begingroup$ @mkt I think there is a substantial overlap between the questions, so I thought it only natural to flag it, but in hindsight a link would probably be sufficient. To point out the difference: The other questioner is satisfied with an intuitive explanation, whereas I would prefer a more rigorous proof. $\endgroup$
    – Frank Vel
    Aug 31, 2018 at 10:11


Browse other questions tagged or ask your own question.