I was reading one of the answers listed on this previous Stackoverflow question about the importance of sufficient statistics (Generalized Linear Models - What's special about the exponential family?):
Jaynes makes the argument that when you leave the exponential family, your estimators cease to be sufficient statistics. If a statistic is sufficient for a parameter then Pr(t|θ)=Pr(X|θ). Implying that the information in t is the same as in the sample X. Bayesian methods always use all the information in X. Non-Bayesian methods use a statistic. If that statistic contains the same information then the estimator will be no worse.
If the statistic is not sufficient, then it will be noisier than the Bayesian estimate. Bayesian estimators are always admissible statistics. If the distribution is not in the exponential family, then the Bayesian estimator will stochastically dominate it, hence the estimator will be inadmissible.
So if you do not make such an assumption, then you would be better off using a Bayesian model in all circumstances. If that were the case, why would you use an alternative?
I am trying to better understand some of the points raised in this answer:
"Bayesian methods always use all the information in X. Non-Bayesian methods use a statistic." I am not quite sure I understand this point : Suppose we have some univariate data (e.g. height measurements, let's call this X). We assume that these measurements likely come from a Normal Distribution. The Non-Bayesian (i.e. Frequentist) would "use all the data (i.e. information)" to calculate the average of these height measurements. The Bayesian would do the same thing - but would supplement his calculations using a prior (e.g. suppose the Bayesian knows that the average height of a basketball player is 200 cm and normally distributed). Based on this - is it not true that both the Bayesian and the Non-Bayesian use all the information in X?
"If the statistic is not sufficient, then it will be noisier than the Bayesian estimate. " Is there any logic or reference that explains why this statement is true?
" Bayesian estimators are always admissible statistics." In the context of statistics, an estimator is called "admissible" if there is no other estimator that is "better" than this estimator (https://en.wikipedia.org/wiki/Admissible_decision_rule). If this is what is meant by admissibility, how can Bayesian estimators always be sufficient? Surely, if you purposefully choose an absurd prior (e.g. the average height of a basketball player is 300 cm, when no human has ever been recorded to have this height), wouldn't the Frequentist estimator at least have a chance of being admissible, seeing as the Bayesian estimator is now surely inadmissible?
At the end of the day, are there any real "dangers" of using "non-sufficient" statistics? Are there ever any instances where it might even be beneficial to use "non-sufficient" statistics? As I understand, the biggest "danger" of using a "non-sufficient" statistic is the fact that this statistic might not be fully taking advantage of the information available within the data - but does this preclude the statistic from being useful?