1
$\begingroup$

I'm reading about sufficient statistics, and have come across two definitions which seems unrelated, and I'm trying to understand their connection.

The first definition is from Wikipedia

A statistic $t = T(X)$ is sufficient for underlying parameter $\theta$ precisely if the conditional probability distribution of the data $X$, given the statistic $t = T(X)$, does not depend on the parameter $\theta$ $$ \operatorname{Pr}(x \mid t, \theta) = \operatorname{Pr}(x \mid t) $$

Wikipedia also provides the following equivalent statements, which I assume is due to bayes theorem $$ \operatorname{Pr}(\theta \mid t, x) = \operatorname{Pr}(\theta \mid t) \\ \operatorname{Pr}(\theta, x \mid t) = \operatorname{Pr}(\theta \mid t)\operatorname{Pr}(x \mid t) $$

The other is an informal description of the theorem, also from Wikipedia

Roughly, given a set $\mathbf {X}$ of independent identically distributed data conditioned on an unknown parameter $\theta$, a sufficient statistic is a function $T(\mathbf{X})$ whose value contains all the information needed to compute any estimate of the parameter (e.g. a maximum likelihood estimate).

which I would understand as $\mathcal{I}(\theta \mid t,x) = \mathcal{I}(\theta \mid t)$ for some information measure, i.e. that the information of $\theta$ given $t$ is the same as if we were given $t$ and $x$, so $t$ is sufficient.

The page on Fisher Information (Wikipedia) shows that the same Fisher information can be inferred from $X$ and $T(X)$, but Fisher information is not mentioned on the page on sufficient statistics. It is not clear if this is the only formal definition of information, or if this holds for other kinds of information.


As there is already a question on the intuition behind the equivalence of the statements, an answer to this question should contain the answer to:

  • How is information defined in this context? Is it Fisher information or something more general?
  • How may one demonstrate using the definition of information and the definition of sufficient statistic, that the two statements are equivalent? (or not?)

A proof of equivalence should preferably be as rigorous as possible, but not incomprehensible.

$\endgroup$
  • 1
    $\begingroup$ Why the downvote? If there's anything I can do to improve the question, I'd like to know. $\endgroup$ – Frank Vel Aug 29 '18 at 21:10
  • $\begingroup$ The duplicate question does not seem to have an accepted answer. The only answers are based on intuition and are not rigorous, which would be preferable. $\endgroup$ – Frank Vel Aug 30 '18 at 9:55
  • 2
    $\begingroup$ Because Wikipedia does not use "all the information needed" in any rigorous or mathematically defined sense that goes beyond the answers in the duplicate, it's fruitless to look for anything more. It's immaterial whether the duplicate has an accepted answer. Indeed, the highest voted answer was posted by the original proposer! $\endgroup$ – whuber Aug 30 '18 at 19:15
  • 1
    $\begingroup$ I'm confused. @FrankVel, did you flag your own question as a duplicate? And then argue against it being a duplicate? $\endgroup$ – mkt - Reinstate Monica Aug 31 '18 at 7:42
  • 1
    $\begingroup$ @mkt I think there is a substantial overlap between the questions, so I thought it only natural to flag it, but in hindsight a link would probably be sufficient. To point out the difference: The other questioner is satisfied with an intuitive explanation, whereas I would prefer a more rigorous proof. $\endgroup$ – Frank Vel Aug 31 '18 at 10:11