I'm reading about sufficient statistics, and have come across two definitions which seems unrelated, and I'm trying to understand their connection.
The first definition is from Wikipedia
A statistic $t = T(X)$ is sufficient for underlying parameter $\theta$ precisely if the conditional probability distribution of the data $X$, given the statistic $t = T(X)$, does not depend on the parameter $\theta$ $$ \operatorname{Pr}(x \mid t, \theta) = \operatorname{Pr}(x \mid t) $$
Wikipedia also provides the following equivalent statements, which I assume is due to bayes theorem $$ \operatorname{Pr}(\theta \mid t, x) = \operatorname{Pr}(\theta \mid t) \\ \operatorname{Pr}(\theta, x \mid t) = \operatorname{Pr}(\theta \mid t)\operatorname{Pr}(x \mid t) $$
The other is an informal description of the theorem, also from Wikipedia
Roughly, given a set $\mathbf {X}$ of independent identically distributed data conditioned on an unknown parameter $\theta$, a sufficient statistic is a function $T(\mathbf{X})$ whose value contains all the information needed to compute any estimate of the parameter (e.g. a maximum likelihood estimate).
which I would understand as $\mathcal{I}(\theta \mid t,x) = \mathcal{I}(\theta \mid t)$ for some information measure, i.e. that the information of $\theta$ given $t$ is the same as if we were given $t$ and $x$, so $t$ is sufficient.
The page on Fisher Information (Wikipedia) shows that the same Fisher information can be inferred from $X$ and $T(X)$, but Fisher information is not mentioned on the page on sufficient statistics. It is not clear if this is the only formal definition of information, or if this holds for other kinds of information.
As there is already a question on the intuition behind the equivalence of the statements, an answer to this question should contain the answer to:
- How is information defined in this context? Is it Fisher information or something more general?
- How may one demonstrate using the definition of information and the definition of sufficient statistic, that the two statements are equivalent? (or not?)
A proof of equivalence should preferably be as rigorous as possible, but not incomprehensible.
