Can someone please explain sufficient statistics in very basic terms? I come from an engineering background, and I have gone through a lot of stuff but failed to find an intuitive explanation.
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
A sufficient statistic summarizes all the information contained in a sample so that you would make the same parameter estimate whether we gave you the sample or just the statistic itself. It's reduction of the data without information loss.
Here's one example. Suppose $X$ has a symmetric distribution about zero. Instead of giving you a sample, I hand you a sample of absolute values instead (that's the statistic). You don't get to see the sign. But you know that the distribution is symmetric, so for a given value $x$, $-x$ and $x$ are equally likely (the conditional probability is $0.5$). So you can flip a fair coin. If it comes up heads, make that $x$ negative. If tails, make it positive. This gives you a sample from $X'$, which has the same distribution as the original data $X$. You basically were able to reconstruct the data from the statistic. That's what makes it sufficient.
In Bayesian terms, you have some observable property $X$ and a parameter $\Theta$. The joint distribution for $X,\Theta$ is specified, but factored as the conditional distribution of $X\mid \Theta$ and the prior distribution of $\Theta$. A statistic $T$ is sufficient for this model if and only if the posterior distribution of $\Theta\mid X$ is the same as that of $\Theta\mid T(X)$, for every prior distribution of $\Theta$. In words, your updated opinion about $\Theta$ after knowing the value of $X$ is the same as your updated opinion about $\Theta$ after knowing the value of $T(X)$, whatever prior opinion you have about $\Theta$. Keep in mind that sufficiency is a model dependent concept.