What are the "Dangers" of using "Non-Sufficient" Statistics?

Question

I was reading one of the answers listed on this previous Stackoverflow question about the importance of sufficient statistics (Generalized Linear Models - What's special about the exponential family?):

Jaynes makes the argument that when you leave the exponential family, your estimators cease to be sufficient statistics. If a statistic is sufficient for a parameter then Pr(t|θ)=Pr(X|θ). Implying that the information in t is the same as in the sample X. Bayesian methods always use all the information in X. Non-Bayesian methods use a statistic. If that statistic contains the same information then the estimator will be no worse.

If the statistic is not sufficient, then it will be noisier than the Bayesian estimate. Bayesian estimators are always admissible statistics. If the distribution is not in the exponential family, then the Bayesian estimator will stochastically dominate it, hence the estimator will be inadmissible.

So if you do not make such an assumption, then you would be better off using a Bayesian model in all circumstances. If that were the case, why would you use an alternative?

I am trying to better understand some of the points raised in this answer:

• "Bayesian methods always use all the information in X. Non-Bayesian methods use a statistic." I am not quite sure I understand this point : Suppose we have some univariate data (e.g. height measurements, let's call this X). We assume that these measurements likely come from a Normal Distribution. The Non-Bayesian (i.e. Frequentist) would "use all the data (i.e. information)" to calculate the average of these height measurements. The Bayesian would do the same thing - but would supplement his calculations using a prior (e.g. suppose the Bayesian knows that the average height of a basketball player is 200 cm and normally distributed). Based on this - is it not true that both the Bayesian and the Non-Bayesian use all the information in X?

• "If the statistic is not sufficient, then it will be noisier than the Bayesian estimate. " Is there any logic or reference that explains why this statement is true?

• " Bayesian estimators are always admissible statistics." In the context of statistics, an estimator is called "admissible" if there is no other estimator that is "better" than this estimator (https://en.wikipedia.org/wiki/Admissible_decision_rule). If this is what is meant by admissibility, how can Bayesian estimators always be sufficient? Surely, if you purposefully choose an absurd prior (e.g. the average height of a basketball player is 300 cm, when no human has ever been recorded to have this height), wouldn't the Frequentist estimator at least have a chance of being admissible, seeing as the Bayesian estimator is now surely inadmissible?

At the end of the day, are there any real "dangers" of using "non-sufficient" statistics? Are there ever any instances where it might even be beneficial to use "non-sufficient" statistics? As I understand, the biggest "danger" of using a "non-sufficient" statistic is the fact that this statistic might not be fully taking advantage of the information available within the data - but does this preclude the statistic from being useful?

Answer 1

The suggestion that Bayesian methods use more information from the data than the sufficient statistic is false. For any Bayesian model with data vector $\mathbf{x}$ and sufficient statistic $\mathbf{T}(\mathbf{x})$ you can use the Fisher-Neyman factorisation for the sufficient statistic to get the posterior form:

$$\begin{align} \pi(\theta | \mathbf{x}) &\propto \pi_0(\theta) L_\mathbf{x}(\theta) \\[6pt] &= \pi_0(\theta) h(\mathbf{x}) g_\theta(\mathbf{T}(\mathbf{x})) \\[6pt] &\propto \pi_0(\theta) g_\theta(\mathbf{T}(\mathbf{x})). \\[6pt] \end{align}$$

As you can see, the posterior distributon is a function of $\mathbf{x}$ only through the sufficient statistic $\mathbf{T}(\mathbf{x})$. This is something that happens in every Bayesian model, and it is really just a manifestation of the likelihood principle. Contrary to the quotation given in your answer, it is classical methods (not Bayesian methods) that sometimes disobey the likelihood principle and use inference procedures that have regard to other aspects of the data outside of the sufficient statistic. Of course, when you move outside parametric models, things become more complicated.

The secondary suggestion that a non-sufficient statistic is noisier than a sufficient statistic is not true in general. Indeed, in any parametric model, the statistic $T=3$ exists and is non-sufficient and it has no noise whatsoever (it is constant so has a variance of zero). Perhaps here the author has in mind some particular sub-class of non-sufficient statistics, but they are not specified, so I would simply ignore this assertion unless you think you can intuite the sub-class of statistics they have in mind.

Finally, the assertion that Bayesian estimators are always admissible is true so long as the model is correctly specified. This is a well-known and desirable property of Bayesian estimators. You can find a discussion of this topic in Robert (2007) (Ch. 8, pp. 391-426) if you would like to know more.