Sufficient Statistics - Relating the Intuition with the Mathematical Definition

I believe the heuristic definition of a Sufficient Statistic makes sense to me - when you take a sample in order to make an inference about the parameter related to the probability distribution, and you reduce that realised sample to a (sufficient) statistic - you do not lose any information or "inference power" regarding the parameter. (However, we still do not know what the actual true value of the parameter is, but I could make an estimate of the range of values that the parameter may be in just as well with the statistic as I could with the original sample).

(Please correct me if you feel there is anything inaccurate about what I said above)

However where I am struggling to connect the dots is how the mathematical definition of Sufficient statistics is saying same thing.

Definition: A statistic $$t = T(X)$$ is sufficient for underlying parameter $$θ$$ precisely if the conditional probability distribution of the data $$X$$, given the statistic $$t = T(X)$$, does not depend on the parameter $$θ$$. (Wikipedia)

Why does $$\theta$$ disappearing in the conditional probability distribution mean the same thing as $$T(X)=t$$ and $$X=x$$ having the same "inference power" about $$\theta$$.

The intuition I have interpreted about this is "Knowing $$T(X)$$ means you know everything about $$\theta$$ from the information supplied by the sample $$X$$" - but my confusion is why does this mean that the conditional distribution should not have $$\theta$$ once we have $$T(X)$$ -> it's not like we know the actual value of $$\theta$$ when we obtain $$T(X)$$, right? I just imagined $$T(X)$$ is all that is required when we try to make an inference - a "guess" about the range of values $$\theta$$ may be in. But what does this have to do with the conditional probability being independent of $$\theta$$?

• Some intuition: The conditional distribution not depending on the parameter $\theta$ means that regardless of the parameter value $\theta$, the conditional distribution will be always the same. In other words, you get the same conditional distribution if $\theta = 1$ or if $\theta = -1$ (or for that sake any other value). Would you be able to make any inference regarding $\theta$ using this conditional distribution after observing a sample? Clearly not. So, the conditional distribution contains no information, so to say, regarding the value of $\theta$ that has generated the data. Commented Feb 21, 2020 at 5:08
• Thanks @baruuum! That clarifies the idea of the conditional distribution being independent of $\theta$, yet I still have trouble seeing the connection between independence of $\theta$ and preserving all information about $\theta$? Commented Feb 21, 2020 at 9:52
• Commented Feb 21, 2020 at 13:36

I think a common way to motivate the mathematical definition is the following.

Say you have the sufficient statistic $$T(X)$$, and I just have the data/random sample $$X$$. By the mathematical definition of sufficiency, $$p(X|T(X), \theta) = P(X|T(X)).$$ The r.h.s. is a probability distribution you, again per definition and in theory, know, and can use to do inference. In particular, if you draw a sample $$X' \sim P(X'|T(X))$$, then this sample is also a draw from the distribution of interest, in particular, the distribution that depends directly on $$\theta$$. But you have now recovered a sample using just the sufficient statistic that contains as much information as the original sample $$X$$. Thus just using your sufficient statistic you can infer just as much (confidence intervals, for example) as me who enjoyed access to the full sample $$X$$, and it is therefore sufficient to know $$T(X)$$.

• Thanks for the answer! I wanted to clarify something. So it seems to me that "$P(X|T(X),\theta) = P(X|T(X))$" is saying "Once you know $T(X)$, I know what the probability distribution of $X$ is (since there is no more unknown $\theta$)." So the distribution of $X$ is unknown due to $\theta$, but when we find out $T(X)$, then the distribution of $X$ becomes known? This feels strange to me because based on one sample (turned into a statistic), we managed to get the distribution of $X$? Commented Feb 21, 2020 at 12:35
• Also, in the last sentence of the solution in: online.stat.psu.edu/stat414/node/282 They claim: "We have just shown that the conditional distribution of $X_1, X_2, ..., X_n$ given $Y$ does not depend on $p$. Therefore, $Y$ is indeed sufficient for $p$. (-HERE->) That is, once the value of $Y$ is known, no other function of $X_1, X_2, ..., X_n$ will provide any additional information about the possible value of $p$." Commented Feb 21, 2020 at 12:48
• The place marked "(-HERE->)" is where I lose the link in understanding Commented Feb 21, 2020 at 12:49
• Two facts: $\theta$ is unknown, while the distribution of the data is not unknown - it is the statistical model we have defined, we call it $p(X|\theta)$. The definition states that, given a sample $X \sim p(X|\theta)$ and a sufficient statistic calculated from that sample, $T(X) = t$, the equation $p(X|T(X) = t, \theta) = p(X|T(X) = t)$ holds. It states nothing about $\theta$ being known. In the example you link to, they explicitly show that, given $T(X)$ (the sample mean), the sufficiency condition holds - namely, $\theta (=p)$ is gone from $p(X|T(X)=t, \theta)$. Commented Feb 21, 2020 at 13:46

"A statistic t=T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ."

If the sampling distribution for some data $$X$$ does not depend on $$\theta$$ then how can that data say anything about $$\theta$$?

It would be like estimating some value by observing something irrelevant (that does not depend on the value to be estimated).

This is a general statement. In this case we have more specifically as data 'the rest of the data conditional on the sufficient statistic'. And that is confusing because the sample distribution for the rest of the data does depend on the parameter to be estimated. It is only that the conditional distribution of that data does not depend on the parameter to estimated.

example 3 (different outcomes of data, but with the same probability for a given $$\theta$$)

(edit: based on the comments I came up with a much more simple/intuitive explanation)

Say you do a urn problem trying to estimate the fraction of blue balls in an urn. You perform an experiment by drawing balls with replacement.

Say you got "$$x_1 = red$$, $$x_2 = blue$$, $$x_3 = red$$, $$x_4 = blue$$, $$x_5 = blue$$"

That is 3 blue balls in total (the total is the sufficient statistic). You could base on this a point estimate of 0.6 for the fraction of blue balls in the urn. (in reality you should take a bigger sample if you want to make a confidence interval with a narrow bandwith, but that makes this example difficult to write down)

Now, does it matter (for the fraction) which particular balls $$x_i$$ where blue (beyond the fact that we already know the total number 3)? Is the estimate gonna be different for "$$x_1 = blue$$, $$x_2 = red$$, $$x_3 = red$$, $$x_4 = blue$$, $$x_5 = blue$$" or any other different observation that also has 3 blue balls in total? Each of these outcomes, with a total of 3 blue balls, are equally possible. So they will not give more information about the fraction of balls in the urn.

We could tabulate all the different outcomes and how the probability of observing them depends on $$\theta$$ (the fraction of blue in the vase)

observation        probability of observing given theta

bbbbb              (1-theta)^0(theta)^5

rbbbb              (1-theta)^1(theta)^4
brbbb              (1-theta)^1(theta)^4
bbrbb              (1-theta)^1(theta)^4
bbbrb              (1-theta)^1(theta)^4
bbbbr              (1-theta)^1(theta)^4

rrbbb              (1-theta)^2(theta)^3
rbrbb              (1-theta)^2(theta)^3
rbbrb              (1-theta)^2(theta)^3
rbbbr              (1-theta)^2(theta)^3
brrbb              (1-theta)^2(theta)^3
brbrb              (1-theta)^2(theta)^3
brbbr              (1-theta)^2(theta)^3
bbrrb              (1-theta)^2(theta)^3
bbrbr              (1-theta)^2(theta)^3
bbbrr              (1-theta)^2(theta)^3

rrrbb              (1-theta)^3(theta)^2
rrbrb              (1-theta)^3(theta)^2
rbrrb              (1-theta)^3(theta)^2
brrrb              (1-theta)^3(theta)^2
rrbbr              (1-theta)^3(theta)^2
rbrbr              (1-theta)^3(theta)^2
brrbr              (1-theta)^3(theta)^2
rbbrr              (1-theta)^3(theta)^2
brbrr              (1-theta)^3(theta)^2
bbrrr              (1-theta)^3(theta)^2

brrrr              (1-theta)^4(theta)^1
rbrrr              (1-theta)^4(theta)^1
rrbrr              (1-theta)^4(theta)^1
rrrbr              (1-theta)^4(theta)^1
rrrrb              (1-theta)^4(theta)^1

rrrrr              (1-theta)^5(theta)^0


Notice that in this table above there are groups of potential observations/outcomes for which the probability to be observed have exactly the same dependency on $$\theta$$. This means that it doesn't matter whether you observe rbrbb or brrbb, they relate to $$\theta$$ in the same way. All the observations with three blue balls can be considered to provide the same information about $$\theta$$.

This is sort of what the sufficient statistic does. It groups together the observations whose Likelihood dependency on $$\theta$$ is the same.

I have deleted examples 1 and 2, because it makes the post very large, but you can still see them in the history of this post

• Thanks for your response! Could you please guide me through this, I'm getting a little lost through this process: - We consider a sample $\mathbf{X} = (X_1, X_2,...,X_n)$ from a population. $X_i \sim f(x_i;\theta)$, where $\theta$ is non-random but unknown - The realised sample becomes $\mathbf{x} = (x_1, x_2,...,x_n)$. This sample can be used to form a inference (e.g. confidence interval) for $\theta$ - We claim that a sufficient statistic $t =T(\mathbf{x})$ can be used to form the same sample inference (e.g. confidence interval) for $\theta$ without $\mathbf{x}$ Commented Feb 21, 2020 at 10:25
• I'm not sure how this relates to the conditional distribution (is it $f(x_{i}; \theta | T(\mathbf{x}) = t)$ ?) Commented Feb 21, 2020 at 10:27
• @user523384 in the first example the sufficient statistic is the maximum of the points. That is $X_{max}$ is the largest from the $(X_1,X_2,...,X_n)$. How the other $X_i$ are distributed can be seen as fully dependent on this maximum $X_{max}$. Commented Feb 21, 2020 at 10:46
• We could say $T \sim f(t;\theta)$ and $X_{i} \sim g(x_i;T)$. Commented Feb 21, 2020 at 10:48
• It is not groundbreaking. It is just that a summary of the data, the statistic $T(x)$, may be sufficient. Different values of $x$ can give the same value of $T(x)$, but how these values $x$ are distributed for a given value of $T(x)$ is independent from $\theta$. Commented Feb 21, 2020 at 10:57

Here is a very simple example that might make things clear. For a normal model $$X\sim N(\theta,1)$$ with a sample size of $$n=1$$, $$X$$ is sufficient for $$\theta$$. Without conditioning on $$T(X)=X$$, then $$P(X\le x)=\Phi(x-\theta)$$ depends on $$\theta$$. If I condition on $$T(X)=X=c$$ then $$P(X\le x|X=c)=\left\{\begin{array}{ll} 0 & \text{if } x < c\\ 1 & \text{if } x \ge c \end{array} \right.$$ does not depend on $$\theta$$. This shows that once $$T(X)$$ is known there is no other information available for estimating $$\theta$$. If there was additional information available then $$P(X\le x|T(X)=c)$$ would depend on $$\theta$$ just like $$P(X\le x)$$ did (and certainly would not be degenerate). Let me known if I have made any mistakes.