# Sufficiency of the sample

Given a sample $X_1,.....,X_n$ with common pdf $f(x;\theta)$, it is clear that the sample $\boldsymbol X$ itself is a sufficient statistic for $\theta$.

I am aware of the meaning of "containing all the information about $\theta$" that is the main feature of a sufficient statistic.

My question is: why is it so crucial to find a sufficient statistic that is "shorter" than the sample? What would be the problems related to the use of the sample as a sufficient statistic in statistical inference?

I hope I am being more clear.

• – Xi'an Jan 3 '16 at 15:52
• For one thing, if a one dimensional statistic is sufficient, you can summarise the amount of information contained in a sample of $n$ points, whatever $n$ is, with a single number. Major gain in storage. For another thing, the sufficient statistic is less "noisy" than the original sample, given that the sample conditional on the sufficient statistic is pure noise and no information. – Xi'an Jan 3 '16 at 15:55
• @Xi'an Maybe I should better explain my doubt. My question is similar but not exactly the same of the one you linked above – PhDing Jan 3 '16 at 16:09
• According to Fisher (who as you probably know introduced the concept of sufficiency) data reduction is essentially the whole purpose of statistics. – dsaxton Jan 3 '16 at 19:02

First, consider the idea of "data reduction." In general, if we can summarize our data using fewer numbers, this may be preferable to using the complete data. If we want to estimate the mean of a population, we can report our sample mean and we lose no information. (One could argue that the sample variance should also be reported - we could report both and we're still only reporting two numbers rather than our entire data set.) This allows for ease of reporting. If you're writing a paper for publication or presenting to a company, it isn't feasible to show all of your data, but the sample mean will be able to convey a the same amount of information regarding the population mean as all of your data would.

In statistical inference, minimal sufficient statistics (sufficient statistics that cannot be reduced any further) have compelling properties that may make further inference easier. For example, minimal sufficient statistics have relationships with complete statistics and conclusions can be drawn from these relationships more easily than directly proving, for example, that a statistic is complete or minimal sufficient. I would refer to a more theoretical resource (i.e. Casella & Berger's Statistical Inference or online resources) that more explicitly detail the benefits of minimal sufficiency.

• Ok, so it is both a matter of clarity and a matter of practical confort. Thus, it is more something artificial than something statistical? – PhDing Jan 3 '16 at 18:27
• Well, I think that it's something statistical in the sense that minimal sufficiency can be a stepping stone to other qualities (i.e. completeness, independence) where it might be more difficult to prove such things directly. As such, minimal sufficiency is certainly something statistically valuable. However, you're also correct in saying that it's a matter of clarity and practical comfort. – Matt Brems Jan 3 '16 at 18:29

In completion to the previous answer, a word of warning: it is only within a very limited class of probability distributions that there exist sufficient statistics that allow for a massive reduction in dimension.

This constraint is known as the Pitman-Koopman-Darmois lemma and is expressed as follows on Wikipedia:

According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter $\theta$ being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases. Less tersely, suppose the $X_i,\ i = 1, \dots, n$ are independent identically distributed random variables whose distribution is known to be in some family of probability distributions with parameter $\theta$. Only if that family is an exponential family is there a (possibly vector-valued) sufficient statistic $T(X_1, \dots, X_n)$ whose number of scalar components does not increase as the sample size $n$ increases.

This result implies that outside exponential families (like the Normal, Poisson, Binomial distributions), there is no possibility of a reduction in dimension from the $n$ original observations to a sufficient statistics. Most often, the "only" reductive sufficient statistic is the order statistic $(X_{(1)},\ldots,X_{(n)})$, where the observations are ordered. This is of the same dimension as the data, the only noise removed being the vector of the ranks.

• Wow, I didn't know the result you posted! Thank you! It is an interesting completion! – PhDing Jan 3 '16 at 18:29
• "...no possibility of reduction" implicitly equates "does not remain bounded" with "equal to $n$." At least in principle there is an enormous middle ground! Persi Diaconis offers a good (but brief) discussion of this issue in Sufficiency as Statistical Symmetry (Proc. AMS Centennial Symposium August 8-12 1988, p. 19). – whuber Jan 3 '16 at 20:07