4
$\begingroup$

Given a sample $X_1,.....,X_n$ with common pdf $f(x;\theta)$, it is clear that the sample $\boldsymbol X$ itself is a sufficient statistic for $\theta$.

I am aware of the meaning of "containing all the information about $\theta$" that is the main feature of a sufficient statistic.

My question is: why is it so crucial to find a sufficient statistic that is "shorter" than the sample? What would be the problems related to the use of the sample as a sufficient statistic in statistical inference?

I hope I am being more clear.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Possible duplicate of Why a sufficient statistic contains all the information needed to compute any estimate of the parameter? $\endgroup$
    – Xi'an
    Jan 3, 2016 at 15:52
  • $\begingroup$ For one thing, if a one dimensional statistic is sufficient, you can summarise the amount of information contained in a sample of $n$ points, whatever $n$ is, with a single number. Major gain in storage. For another thing, the sufficient statistic is less "noisy" than the original sample, given that the sample conditional on the sufficient statistic is pure noise and no information. $\endgroup$
    – Xi'an
    Jan 3, 2016 at 15:55
  • $\begingroup$ @Xi'an Maybe I should better explain my doubt. My question is similar but not exactly the same of the one you linked above $\endgroup$
    – PhDing
    Jan 3, 2016 at 16:09
  • $\begingroup$ According to Fisher (who as you probably know introduced the concept of sufficiency) data reduction is essentially the whole purpose of statistics. $\endgroup$
    – dsaxton
    Jan 3, 2016 at 19:02

2 Answers 2

Reset to default
2
$\begingroup$

There are a couple of answers to your question.

First, consider the idea of "data reduction." In general, if we can summarize our data using fewer numbers, this may be preferable to using the complete data. If we want to estimate the mean of a population, we can report our sample mean and we lose no information. (One could argue that the sample variance should also be reported - we could report both and we're still only reporting two numbers rather than our entire data set.) This allows for ease of reporting. If you're writing a paper for publication or presenting to a company, it isn't feasible to show all of your data, but the sample mean will be able to convey a the same amount of information regarding the population mean as all of your data would.

In statistical inference, minimal sufficient statistics (sufficient statistics that cannot be reduced any further) have compelling properties that may make further inference easier. For example, minimal sufficient statistics have relationships with complete statistics and conclusions can be drawn from these relationships more easily than directly proving, for example, that a statistic is complete or minimal sufficient. I would refer to a more theoretical resource (i.e. Casella & Berger's Statistical Inference or online resources) that more explicitly detail the benefits of minimal sufficiency.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ok, so it is both a matter of clarity and a matter of practical confort. Thus, it is more something artificial than something statistical? $\endgroup$
    – PhDing
    Jan 3, 2016 at 18:27
  • 2
    $\begingroup$ Well, I think that it's something statistical in the sense that minimal sufficiency can be a stepping stone to other qualities (i.e. completeness, independence) where it might be more difficult to prove such things directly. As such, minimal sufficiency is certainly something statistically valuable. However, you're also correct in saying that it's a matter of clarity and practical comfort. $\endgroup$
    – Matt Brems
    Jan 3, 2016 at 18:29
8
$\begingroup$

In completion to the previous answer, a word of warning: it is only within a very limited class of probability distributions that there exist sufficient statistics that allow for a massive reduction in dimension.

This constraint is known as the Pitman-Koopman-Darmois lemma and is expressed as follows on Wikipedia:

According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter $\theta$ being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases. Less tersely, suppose the $X_i,\ i = 1, \dots, n$ are independent identically distributed random variables whose distribution is known to be in some family of probability distributions with parameter $\theta$. Only if that family is an exponential family is there a (possibly vector-valued) sufficient statistic $T(X_1, \dots, X_n)$ whose number of scalar components does not increase as the sample size $n$ increases.

This result implies that outside exponential families (like the Normal, Poisson, Binomial distributions), there is no possibility of a reduction in dimension from the $n$ original observations to a sufficient statistics. Most often, the "only" reductive sufficient statistic is the order statistic $(X_{(1)},\ldots,X_{(n)})$, where the observations are ordered. This is of the same dimension as the data, the only noise removed being the vector of the ranks.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Wow, I didn't know the result you posted! Thank you! It is an interesting completion! $\endgroup$
    – PhDing
    Jan 3, 2016 at 18:29
  • 5
    $\begingroup$ "...no possibility of reduction" implicitly equates "does not remain bounded" with "equal to $n$." At least in principle there is an enormous middle ground! Persi Diaconis offers a good (but brief) discussion of this issue in Sufficiency as Statistical Symmetry (Proc. AMS Centennial Symposium August 8-12 1988, p. 19). $\endgroup$
    – whuber ♦
    Jan 3, 2016 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.