Sufficient statistics for layman

Question

Can someone please explain sufficient statistics in very basic terms? I come from an engineering background, and I have gone through a lot of stuff but failed to find an intuitive explanation.

Answer 1

A sufficient statistic summarizes all the information contained in a sample so that you would make the same parameter estimate whether we gave you the sample or just the statistic itself. It's reduction of the data without information loss.

Here's one example. Suppose $X$ has a symmetric distribution about zero. Instead of giving you a sample, I hand you a sample of absolute values instead (that's the statistic). You don't get to see the sign. But you know that the distribution is symmetric, so for a given value $x$, $-x$ and $x$ are equally likely (the conditional probability is $0.5$). So you can flip a fair coin. If it comes up heads, make that $x$ negative. If tails, make it positive. This gives you a sample from $X'$, which has the same distribution as the original data $X$. You basically were able to reconstruct the data from the statistic. That's what makes it sufficient.

Answer 2

In Bayesian terms, you have some observable property $X$ and a parameter $\Theta$. The joint distribution for $X,\Theta$ is specified, but factored as the conditional distribution of $X\mid \Theta$ and the prior distribution of $\Theta$. A statistic $T$ is sufficient for this model if and only if the posterior distribution of $\Theta\mid X$ is the same as that of $\Theta\mid T(X)$, for every prior distribution of $\Theta$. In words, your updated uncertainty about $\Theta$ after knowing the value of $X$ is the same as your updated uncertainty about $\Theta$ after knowing the value of $T(X)$, whatever prior information you have about $\Theta$. Keep in mind that sufficiency is a model dependent concept.

Somewhat surprisingly, this Bayesian definition of sufficiency is due to Kolmogorov (see the second paragraph on page 1012 of Rikhin (1990)).

Answer 3

Say you have a coin, and you don't know whether it's fair or not. In other words, it has probability $p$ of coming up heads ($H$) and $1 - p$ of coming up tails ($T$), and you don't know the value of $p$.

You try to get an idea of the value of $p$ by tossing the coin several times, say $n$ times.

Let's say $n = 5$ and the outcome you happen to get is the sequence $(H, H, T, H, T)$.

Now you want your statistician friend to estimate the value of $p$ for you, and perhaps tell you if the coin is likely to be fair or not. What information do you need to tell them so that they can do their calculations and make their conclusions?

You could tell them all of the data, i.e. $(H, H, T, H, T)$. Is this necessary though? Could you summarise this data without losing any relevant information?

It is clear that the order of the coin tosses is irrelevant, because you were doing the same thing for each coin toss, and the coin tosses didn't influence each other. If the outcome were $(H, H, T, T, H)$ instead, for example, our conclusions won't be any different. It follows that all you really need to tell your statistician friend is the count of how many heads there were.

We express this by saying the number of heads is a sufficient statistic for p.

This example gives the flavour of the concept. Read on if you'd like to see how it connects with the formal definition.

Formally, a statistic is sufficient for a parameter if, given the value of the statistic, the probability distribution of the outcomes doesn't involve the parameter.

In this example, before we know the number of heads, the probability of any outcome is $p^\text{number of heads}(1 - p)^\text{n - number of heads}$. Obviously this depends on $p$.

But once we know that the number of heads is 3 (or any other value), all the outcomes with 3 heads ($(H, H, T, H, T)$, $(H, H, T, T, H)$, $...$) are equally likely (in fact there are ten possibilities so they all have probability $1/10$). So the conditional distribution of the outcomes no longer has anything to do with $p$. Intuitively this means whichever specific outcome we observe won't tell us any more information about $p$, because the outcomes aren't affected by $p$.

As an aside, note that the probability before we know the number of heads only depends on $p$ through the $\text{number of heads}$. It turns out that this is equivalent to the $\text{number of heads}$ being sufficient for $p$.