Bayesian models and exchangeability

Question

I did not see that explicitly mentioned, even though I think it is correct.

Isn't the exchangeability assumption the most common assumption about examples in the Bayesian setting?

I am thinking of a model of the form $p(x_1,\ldots,x_n,\theta) = p(\theta) \prod_{i=1}^n p(x_i \mid \theta)$, where $p(\theta)$ is a prior. By deFinetti's theorem, I think this means that the $x_i$ exchangeable.

Having a prior this way is very common in the Bayesian setting. Hence my conclusion. Is there anything wrong with my reasoning?

Answer 1

You're right but:

• More precisely, we should say that $X_1$, $\ldots,$, $X_n$ are exchangeable under the prior predictive distribution (as well as the posterior)

  • This fact is elementary (conditionally i.i.d. $\implies$ exchangeability), it does not stem from deFinetti's theorem (this theorem claims that exchangeability implies conditionally i.i.d. for an infinite sequence $(X_1, \ldots, X_n, \ldots)$).

Answer 2

There are a few points worth noting here:

• (IID $\implies$ exchangeability): The conditional IID form immediately implies exchangeability of the values. This does not require de Finetti's representation theorem. Stéphane Laurent is right to characterise this as an elementary result (proof below).

• (IID $\impliedby$ infinite exchangeability): De Finetti's representation theorem (and its extension by Hewitt and Savage) show that exchangeability of an infinite sequence implies the conditional IID form. Finite exchangeability is not sufficient to give the conditional IID form, but there are some results showing that it comes close (i.e., finite exchangeability is sufficient to show that the true probabilities pertaining to a set of values are within a particular bound of the conditional IID form).

• These results do not require a prior: Both of the above results hold, inter alia, with respect to the sampling distribution of the observable values in the problem, and so they hold without any specification of a prior distribution. Indeed, you do not even have to be working within the Bayesian paradigm at all for these results to be applicable (see O'Neill 2009 for further discussion on this issue).

---

THEOREM: If $X_1,...,X_n$ are IID conditional on $\theta$ then they are exchangeable.

PROOF: Choose and arbitrary permutation $\pi$ on the set $\{ 1,...,n \}$. Since the values $X_1,...,X_n$ are IID conditional on $\theta$ we have:

$$\begin{equation} \begin{aligned} \mathbb{P}(X_1 \leqslant x_1,...,X_n \leqslant x_n | \theta) &= \prod_{i=1}^n \mathbb{P}(X_i \leqslant x_i | \theta) \\[6pt] &= \prod_{i=1}^n \mathbb{P}(X_{\pi(i)} \leqslant x_i | \theta) \\[6pt] &= \mathbb{P}(X_{\pi(1)} \leqslant x_1,...,X_{\pi(n)} \leqslant x_n | \theta). \\[6pt] \end{aligned} \end{equation}$$

(The second step above follows simply by taking the product of values in the order of the permutation - it follows from the associativity of multiplication.) We then have:

$$\begin{equation} \begin{aligned} \mathbb{P}(X_1 \leqslant x_1,...,X_n \leqslant x_n) &= \mathbb{E}_\theta \bigg[ \mathbb{P}(X_1 \leqslant x_1,...,X_n \leqslant x_n | \theta) \bigg] \\[6pt] &= \mathbb{E}_\theta \bigg[ \mathbb{P}(X_{\pi(1)} \leqslant x_1,...,X_{\pi(n)} \leqslant x_n | \theta) \bigg] \\[6pt] &= \mathbb{P}(X_{\pi(1)} \leqslant x_1,...,X_{\pi(n)} \leqslant x_n), \\[6pt] \end{aligned} \end{equation}$$

which establishes exchangeability of $X_1,...,X_n$. $\blacksquare$

Answer 3

No, I think your reasoning is right. Exchangeability was a very important property to de Finetti in his development of probability theory (which is Bayesian). It also is important regarding permutation tests. Often in doing statistical inference we assume observations are independent and identically distributed and this of course implies exchangeability.