19
$\begingroup$

I've just started studying statistics and I can't get an intuitive understanding of sufficiency. To be more precise I can't understand how to show that the following two paragraphs are equivalent:

Roughly, given a set X of independent identically distributed data conditioned on an unknown parameter θ, a sufficient statistic is a function T(X) whose value contains all the information needed to compute any estimate of the parameter.

A statistic T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic T(X), does not depend on the parameter θ.

(I've taken the quotes from Sufficient statistic)

Though I understand the second statement, and I can use the factorization theorem to show if a given statistic is sufficient, I can't understand why a statistic with such a property has also the property that it "contains all the information needed to compute any estimate of the parameter". I am not looking for a formal proof, which would help anyway to refine my understanding, I'd like to get an intuitive explanation of why the two statements are equivalent.

To recap, my questions are: why the two statements are equivalent? Could someone provide an intuitive explanation for their equivalence?

$\endgroup$
9
  • 2
    $\begingroup$ The main intuitive idea is that you sometimes don't need to see the whole sample because you can find a statistic that summarises all the information needed from the sample. Take, for example, a binomial distribution: all you need to know for your model is the sum of successes. You don't lose anything of value if I only tell you that $\sum_{i}^{n} x_i = c$, instead of showing you the whole set of sampled values $x = \{1, 0, 0, 1, 0, 1, ... \}$. $\endgroup$
    – mugen
    Jan 18, 2015 at 23:40
  • $\begingroup$ I understand why I would need a sufficient statistics and how to show that the sum of successes is a sufficient statistic for p in a Bernoulli process. What I don't understand is why a statistic like that described in the second paragraph contains all the information needed to compute any estimate of the parameter. $\endgroup$
    – gcoll
    Jan 19, 2015 at 0:00
  • 4
    $\begingroup$ Strictly speaking, the first quotation is just plain wrong. There are plenty of estimators that can be computed from the whole dataset which cannot be computed solely from sufficient statistics. That's one reason the quote begins "roughly." Another reason is that it doesn't supply a quantitative or rigorous definition of "information." Since a much more accurate (but still intuitive) characterization had been given in the preceding paragraph, though, there is little problem with this quotation in the proper context. $\endgroup$
    – whuber
    Jan 19, 2015 at 1:52
  • 1
    $\begingroup$ It has connection to maximum likelihood and it is essentially the information needed in the maximum likelihood $\endgroup$
    – Kamster
    Jan 19, 2015 at 2:56
  • 1
    $\begingroup$ Following the comments of whuber and @Kamster, I probably got a better understanding. When we say that a sufficient statistic contains all the information needed to compute any estimate of the parameter, do we actually mean that it is enough to compute the maximum likelihood estimator (which is a function of all sufficient statistics)? It this is true, the issue was all related to the (non-)definition of "information", as whuber suggested, and my question is answered. $\endgroup$
    – gcoll
    Jan 19, 2015 at 22:07

4 Answers 4

5
$\begingroup$

Following the comments of @whuber and @Kamster, I probably got a better understanding. When we say that a sufficient statistic contains all the information needed to compute any estimate of the parameter, what we actually mean is that it is enough to compute the maximum likelihood estimator (which is a function of all sufficient statistics).

Given that I am answering my own question, and so I am not 100% sure of the answer, I will not mark it as correct until I get some feedback. Please add any comment and down-vote if you think I am being wrong/imprecise/etc...

(Let me know if this is not compatible with SE etiquette, being this my first question I beg your clemency if I am violating any rule)

$\endgroup$
5
$\begingroup$

As I was studying about sufficiency I came across your question because I also wanted to understand the intuition about From what I've gathered this is what I come up with (let me know what you think, if I made any mistakes, etc).

Let $X_1,\ldots,X_n$ be a random sample from a Poisson distribution with mean $\theta>0$.

We know that $T({\bf{X}})=\sum_{i=1}^{n} X_i$ is a sufficient statistic for $\theta$, since the conditional distribution of $X_1,\ldots,X_n$ given $T({\bf{X}})$ is free of $\theta$, in other words, does not depend on $\theta$.

Now, statistician $A$ knows that $X_1,\ldots,X_n \overset{i.i.d}{\sim} Poisson(4)$ and creates $n=400$ random values from this distribution:

    n <- 400
    theta <- 4
    set.seed(1234)
    x <- rpois(n, theta)
    y=sum(x)
       
    freq.x <- table(x) # We will use this latter on
    rel.freq.x <- freq.x/sum(freq.x)

For the values statistician $A$ has created, he takes the sum of it and asks statistician $B$ the following:

"I have these sample values $x_1,\ldots,x_n$ taken from a Poisson distribution. Knowing that $\sum_{i=1}^{n} x_i = y = 4068$, what can you tell me about this distribution?"

So, knowing only that $\sum_{i=1}^{n} x_i = y = 4068$ (and the fact that the sample arose from a Poisson distribution) is sufficient for statistician $B$ to say anything about $\theta$? Since we know that this is a sufficient statistic we know that the answer is "yes".

To gain some intuition about the meaning of this, let's do the following (taken from Hogg & Mckean & Craig's "Introduction to Mathematical Statistics", 7th edition, exercise 7.1.9):

"$B$ decides to create some fake observations, which he calls $z_1,z_2,\ldots,z_n$ (as he knows they will probably not be equal the original $x$-values) as follows. He notes that the conditional probability of independent Poisson random variables $Z_1,Z_2\ldots,Z_n$ being equal to $z_1,z_2,\ldots,z_n$, given $\sum z_i = y$, is

$$\cfrac{\frac{\theta^{z_1}e^{-\theta}}{z_1!} \frac{\theta^{z_2}e^{-\theta}}{z_2!} \cdots \frac{\theta^{z_n}e^{-\theta}}{z_n!}}{\frac{n \theta^{y}e^{-n\theta}}{y!}}=\frac{y!}{z_1!z_2! \cdots z_n!} \left(\frac{1}{n}\right)^{z_1} \left(\frac{1}{n}\right)^{z_2} \cdots \left(\frac{1}{n}\right)^{z_n}$$

since $Y=\sum Z_i$ has a Poisson distribution with mean $n \theta$. The latter distribution is multinomial with $y$ independent trials, each terminating in one of $n$ mutually exclusive and exhaustive ways, each of which has the same probability $1/n$. Accordingly, $B$ runs such a multinomial experiment $y$ independent trials and obtains $z_1,\ldots,z_n$."

This is what the exercise states. So, let's do exactly that:

    # Fake observations from multinomial experiment
    prob <- rep(1/n, n)
    set.seed(1234)
    z <- as.numeric(t(rmultinom(y, n=c(1:n), prob)))
    y.fake <- sum(z) # y and y.fake must be equal
    freq.z <- table(z)
    rel.freq.z <- freq.z/sum(freq.z)

And let's see what $Z$ looks like (I'm also plotting the real density of Poisson(4) for $k=0,1,\ldots,13$ - anything above 13 is pratically zero -, for comparison):

    # Verifying distributions
    k <- 13
    plot(x=c(0:k), y=dpois(c(0:k), lambda=theta, 
         log = FALSE),t="o",ylab="Probability",xlab="k",
         xlim=c(0,k), ylim=c(0, max(c(rel.freq.x, rel.freq.z))))
    lines(rel.freq.z, t="o", col="green", pch=4)
    legend(8,0.2, legend=c("Real Poisson", "Random Z given y"), 
           col = c("black", "green"), pch=c(1,4))

enter image description here

So, knowing nothing about $\theta$ and knowing only the sufficient statistic $Y=\sum X_i$ we were able to recreate a "distribution" that looks a lot like a Poisson(4) distribution (as $n$ increases, the two curves become more similar).

Now, comparing $X$ and $Z|y$:

    plot(rel.freq.x, t="o", pch=16, col="red", 
         ylab="Relative Frequency", xlab="k",
         ylim=c(0, max(c(rel.freq.x, rel.freq.z))))
    lines(rel.freq.z, t="o", col="green", pch=4)
    legend(7, 0.2, legend=c("Random X", "Random Z given y"), 
           col = c("red", "green"), pch=c(16,4))

enter image description here

We see that they are pretty similar, as well (as expected)

So, "for the purpose of making a statistical decision, we can ignore the individual random variables $X_i$ and base the decision entirely on the $Y=X_1+X_2+\cdots+X_n$" (Ash, R. "Statistical Inference: A concise course", page 59).

$\endgroup$
1
$\begingroup$

Let me give another perspective that may help. This is also qualitative, but there is a rigorous version of that particularly important in Information Theory - known as Markov property.

In the beginning, we have two objects, data (coming from a Random Variable, call it X) and parameter, $\theta$ (another rv, implicitly assumed since we are talking about its estimator). These two, are assumed to be dependent (otherwise, there is no point in trying to estimate one from the other). Now, the third object enters the game, Sufficient Statistic, T. The intuitive idea when we say T is enough to estimate $\theta$ really means that if we know T (ie conditioned on T), X provides no additional info, that is, X and $\theta$ are independent. In other word, knowledge of X is equivalent to knowledge of T as far as estimation of $\theta$ is concerned. Note that in probabilities are where all the uncertainties are captured, and hence "any estimate" when (conditional) probabilities are independent (eg conditional densities factorize).

$\endgroup$
1
$\begingroup$

The second sentence in the quote is proven by the factorization theorem--which shows that a sample conditioned on its sufficient statistic is independent of the parameter. The first sentence is equivalent to the second sentence (to the extent that an informal statement can be equivalent to a mathematical statement) if we modify it to say "all the parameter information." In other words, the statement assumes that only parameter (Fisher) information is inferentially useful. Or, at least, it assumes any other type of information, however conceived and defined and measured, must be redundant with the Fisher information for purposes of improving the accuracy and/or precision of an estimate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.