I have read (and re-read) the wikipedia article on Exchangeability https://en.wikipedia.org/wiki/Exchangeable_random_variables . The disconnect for me is that : after having sampled without replacement the distribution of the remaining items differs from the original distribution. I am specifically referring to a small enough population that the sample size may have a meaningful effect on the distribution of the unsampled items in the population.

So let's say we are at the tenth item of a given sample - that distribution also differs from the distribution (and expected succeeding items) of a separate sample from the same population.

So I'm missing the basic intuition of what the Exchangeability means here. A concise explanation specifically explaining why the "without replacement" actually works (vs "with" replacement that instead would actually make sense to me) would be helpful.


1 Answer 1


Consider random sampling without replacement from a finite population of $N$ distinct members (we index them $1,2,\dotsc, N$). Let $X_1$ be the result of first draw, $X_2$ result of second draw, etc. Then we find $$ P(X_1=j)=\frac1{N}, \quad j=1,\dotsc,N. $$ Continuing $P(X_2=j' \mid X_1=j)=\frac1{N-1}, \quad j'\not=j$. But the interest is in the marginal distribution of $X_2$: $$ P(X_2=j') =\sum_{j\not=j'} P(X_2=j'\mid X_1=j)\cdot P(X_1=j)=\sum_{j\not=j'}\frac1{N-1}\cdot \frac1{N} = \\ (N-1)\cdot \frac1{N-1}\cdot\frac1{N}=\frac1{N}. $$ This argument can be continued and extended to bivariate marginals ...

The error in your argument ... the distribution of the remaining items differs from the original distribution is the sequential thinking. But the meaning of the distribution of $(X_1, \dotsc,X_N)$ is its distribution before the sampling starts. In your thinking you refers to the distribution of $X_1$, $X_2 \mid X_1$, $X_3 \mid (X_1,X_2)$ and so on, sequentially. But exchangeability refers to the distribution of $(X_1, \dotsc,X_N)$ before we start sampling. I think this is the intuition you are missing.

Another way of looking is via symmetry (exchangeability is a kind of symmetry). If you draw (for some small $N$, maybe 3) the full probability tree for simple random sampling without replacement, the tree will be fully symmetric. Any permutation of the indices 1,2,3 will leave the tree essentially unchanged.

Finally note that in this example we have only finite exchangeability. The de Finetti representation theorem, for instance, requires infinite exchangeability, but our sequence of course cannot be extended when we have reached $N$ ...

  • 1
    $\begingroup$ OK I think I see now. The point of exchangeability then seems to be "the prior probability of drawing an observation X_k is not a function of the other observations" $\endgroup$ Dec 31, 2018 at 0:38
  • 1
    $\begingroup$ I don't think that's it at all. The point simply is that the joint distribution is perfectly symmetrical: it remains the same under any permutation of the components. Focusing on the marginal distributions may be useful but it misses this basic conceptual point. $\endgroup$
    – whuber
    Dec 31, 2018 at 17:15
  • $\begingroup$ Is it necessary for a finite population of $N$ distinct members to follow a uniform distribution? I.e. all $N$ members are equiprobable as described in the first equation? $\endgroup$
    – ado sar
    Nov 27, 2023 at 17:34
  • $\begingroup$ @adosar: No, it is not necessary, but it is the case treated here $\endgroup$ Nov 27, 2023 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.