# Does the factorization theorem prove that the simplest factorization of the PDF is the most informative?

Let me state the factorization theorem as: the existence of a PDF factorization where $$X$$ depends on the parameter only through $$T(X)$$, proves $$T(X)$$ is sufficient, defined as conveying the maximum possible parameter information from $$X$$. Its proof (or all versions I've found) shows how one can factor a PDF into the simple product of two components, one independent of the parameter and the other depending on the parameter only through $$T(X)$$.

This is just algebra, proving only the statement: "Some PDFs can be factored in this way." The proof then imposes, definitionally, that $$T(X)$$ is sufficient. It seems to rely on an intuitively appealing, but mathematically unjustified, assumption: the information of a PDF's individual factors cannot exceed their collective information. This assumption fails to consider, much less to exclude, another possibility: there may exist a partition of the simplest factorization into many factors, some or all of which individually increase the information of $$T(X)$$ conditionally, but which collectively cancel. Let me illustrate.

Basu (1964) showed for bivariate normal sample $$(X, Y)$$ with population correlation $$\rho$$, $$X$$ and $$Y$$ are separately ancillary but jointly sufficient (though obviously not minimal sufficient). In this particular case, conditional inference is not possible, i.e., neither $$r\mid X$$ nor $$r\mid Y$$ is more informative than sufficient statistic Pearson's $$r$$. But can we prove no such example exists? Shouldn't we have to for the FT to hold?

The upshot is, it's basic algebra to pull factors out of polynomials that otherwise cancel$$-$$say, the product of a factor and its reciprocal, or equal addends of opposite sign. Shouldn't the FT have to refute this possibility? If it doesn't, isn't it just an axiom? Or is there a flaw in my logic? Keep in mind my example is illustrative only: the burden isn't on me to prove a counterexample, but on the FT to prove there can be no counterexample.

The definition of sufficiency doesn't involve information. A statistic $$T(X)$$ is sufficient for a parameter $$\theta$$ if the conditional distribution $$X\mid T(X)$$ does not depend on $$\theta$$.

The factorisation theorem gives a condition for $$T$$ to be sufficient. This is very straightforward for discrete or sufficiently smooth continuous cases, and almost looks trivial, but is quite subtle for more general cases.

Neither the definition nor the factorisation theorem directly addresses the question of whether there might be another way to extract more information. That's a further result or results, and depends on exactly what you mean by information. In that direction we have:

• the Cram'er-Rao variance bound for an estimator computed from $$T(X)$$ is the same as for an estimator computed from $$X$$
• the Rao-Blackwell theorem says that conditioning on a sufficient statistic improves any estimator
• the Lehman-Scheff'e theorem says that all the estimators that attain the Cram'er-Rao variance bound are functions of a minimal sufficient statistics
• likelihood ratios based on $$X$$ and on $$T(X)$$ are the same, so the most powerful test for any point null and alternative is the same
• likelihood ratios based on $$X$$ and on $$T(X)$$ are the same, so the asymptotic efficiency bound in the Convolution Theorem and Local Asymptotic Minimax theorem are the same for both.
• likelihood ratios based on $$X$$ and on $$T(X)$$ are the same, so posterior beliefs about $$\theta$$ based on observing $$T(X)$$ are/should be the same as based on $$X$$
• To be clear, when Wikipedia says "A sufficient statistic contains all of the information that the dataset provides about the model parameters," this is false? Not to say that Wikipedia is the ultimate source for statistics, but the same language appears in many stats texts, and articles going back to Fisher's seminal papers on sufficiency. Has there been a re-evaluation of the implications of sufficiency since his day, which somehow has not penetrated standard sources? If so, you'd think this would be a controversial topic on the Wikipedia talk page and elsewhere... Commented Aug 1 at 23:51
• Wikipedia is a bit imprecise in this case. "All of the information" is the motivation for the concept of sufficiency, but it is not part of the definition or of the factorisation theorem. If "all of the information" were straightforwardly untrue, however, sufficiency wouldn't have been an interesting concept (or it would be defined differently). It has also been re-evaluated a bit since Fisher because so few problems have finite-dimensional sufficient statistics. Commented Aug 2 at 1:00
• My problem is, I can't tell if your interpretation is the consensus view among the broader statistics community, or if it's the view within a particular statistical school. By analogy, Bayesians have mathematical formulations of probability as prior belief; likelihoodists have their counterexamples to Mayo's disproof of the SLP; etc. Maybe mathematical statisticians have re-evaluated, while those in other subfields continue to define the FT as I did above. Can you cite some general authority to the effect that "all the information" is the motivation and not the implication of sufficiency? Commented Aug 2 at 7:54
• Although, possibly the problem is resolved by distinguishing between sufficiency and the Sufficiency Principle. According to the top answer at stats.stackexchange.com/questions/379798/: "The Sufficiency Principle says that we should draw the same conclusions in two experiments where a sufficient statistic has the same value." Then perhaps the factorization theorem and/or the definition of sufficiency as conditional independence of the sample from the parameter, plus the SP, together imply Wikipedia's definition of sufficiency. Mathematical statistics does not encompass all of estimation theory Commented Aug 2 at 8:11
• +1. Mathematical results like these are not a matter of contention, philosophy, consensus, or style among the statistics community: they are the basic, proven facts on which we rely for justification.
– whuber
Commented Aug 2 at 19:21