Sampling from a finite sequence without replacement yields exchangeable sequences?

Question

I have read (and re-read) the wikipedia article on Exchangeability https://en.wikipedia.org/wiki/Exchangeable_random_variables . The disconnect for me is that : after having sampled without replacement the distribution of the remaining items differs from the original distribution. I am specifically referring to a small enough population that the sample size may have a meaningful effect on the distribution of the unsampled items in the population.

So let's say we are at the tenth item of a given sample - that distribution also differs from the distribution (and expected succeeding items) of a separate sample from the same population.

So I'm missing the basic intuition of what the Exchangeability means here. A concise explanation specifically explaining why the "without replacement" actually works (vs "with" replacement that instead would actually make sense to me) would be helpful.

Answer 1

Consider random sampling without replacement from a finite population of $N$ distinct members (we index them $1,2,\dotsc, N$). Let $X_1$ be the result of first draw, $X_2$ result of second draw, etc. Then we find $$ P(X_1=j)=\frac1{N}, \quad j=1,\dotsc,N. $$ Continuing $P(X_2=j' \mid X_1=j)=\frac1{N-1}, \quad j'\not=j$. But the interest is in the marginal distribution of $X_2$: $$ P(X_2=j') =\sum_{j\not=j'} P(X_2=j'\mid X_1=j)\cdot P(X_1=j)=\sum_{j\not=j'}\frac1{N-1}\cdot \frac1{N} = \\ (N-1)\cdot \frac1{N-1}\cdot\frac1{N}=\frac1{N}. $$ This argument can be continued and extended to bivariate marginals ...

The error in your argument ... the distribution of the remaining items differs from the original distribution is the sequential thinking. But the meaning of the distribution of $(X_1, \dotsc,X_N)$ is its distribution before the sampling starts. In your thinking you refers to the distribution of $X_1$, $X_2 \mid X_1$, $X_3 \mid (X_1,X_2)$ and so on, sequentially. But exchangeability refers to the distribution of $(X_1, \dotsc,X_N)$ before we start sampling. I think this is the intuition you are missing.

Another way of looking is via symmetry (exchangeability is a kind of symmetry). If you draw (for some small $N$, maybe 3) the full probability tree for simple random sampling without replacement, the tree will be fully symmetric. Any permutation of the indices 1,2,3 will leave the tree essentially unchanged.

Finally note that in this example we have only finite exchangeability. The de Finetti representation theorem, for instance, requires infinite exchangeability, but our sequence of course cannot be extended when we have reached $N$ ...