# Bayesian models and exchangeability

I did not see that explicitly mentioned, even though I think it is correct.

Isn't the exchangeability assumption the most common assumption about examples in the Bayesian setting?

I am thinking of a model of the form $p(x_1,\ldots,x_n,\theta) = p(\theta) \prod_{i=1}^n p(x_i \mid \theta)$, where $p(\theta)$ is a prior. By deFinetti's theorem, I think this means that the $x_i$ exchangeable.

Having a prior this way is very common in the Bayesian setting. Hence my conclusion. Is there anything wrong with my reasoning?

• I do not think you have to use de Finetti's theorem here. From the expression you have, it follows that $X_1,...,X_n$ are iid and then they are exchangeable. This is, you are assuming something stronger. – user10525 Aug 16 '12 at 18:40
• I am talking about exchanginability for the distribution $p(X_1,\ldots,X_n)$. They are i.i.d. conditioned on $\theta$. – singelton Aug 16 '12 at 19:11
• It is immediate from the iid assumption. See Stéphane Laurent's comment. – user10525 Aug 17 '12 at 8:16

You're right but:

• More precisely, we should say that $X_1$, $\ldots,$, $X_n$ are exchangeable under the prior predictive distribution (as well as the posterior)

• This fact is elementary (conditionally i.i.d. $\implies$ exchangeability), it does not stem from deFinetti's theorem (this theorem claims that exchangeability implies conditionally i.i.d. for an infinite sequence $(X_1, \ldots, X_n, \ldots)$).
• maybe deFinetti's is a heavy gun to use to show exchangeability for $p(X_1,\ldots,X_n)$... why does conditional i.i.d.ness leads to exchangeability immediately? – singelton Aug 16 '12 at 19:42
• @singelton this results from the formula $\Pr(X_1 \in A_1, \ldots, X_n \in A_n)=E[\Pr(X_1 \in A_1, \ldots, X_n \in A_n\mid \Theta)]$. – Stéphane Laurent Aug 16 '12 at 20:20
• @StéphaneLaurent My comment about iid and exchangeability was not meant to be connected to deFinetti's theorem. I just wanted to point out that exchangeability is commonly satisfied in inference problems. – Michael Chernick Aug 16 '12 at 22:26
• @MichaelChernick My answer was for the OP :) – Stéphane Laurent Aug 17 '12 at 5:27

There are a few points worth noting here:

• (IID $$\implies$$ exchangeability): The conditional IID form immediately implies exchangeability of the values. This does not require de Finetti's representation theorem. Stéphane Laurent is right to characterise this as an elementary result (proof below).

• (IID $$\impliedby$$ infinite exchangeability): De Finetti's representation theorem (and its extension by Hewitt and Savage) show that exchangeability of an infinite sequence implies the conditional IID form. Finite exchangeability is not sufficient to give the conditional IID form, but there are some results showing that it comes close (i.e., finite exchangeability is sufficient to show that the true probabilities pertaining to a set of values are within a particular bound of the conditional IID form).

• These results do not require a prior: Both of the above results hold, inter alia, with respect to the sampling distribution of the observable values in the problem, and so they hold without any specification of a prior distribution. Indeed, you do not even have to be working within the Bayesian paradigm at all for these results to be applicable (see O'Neill 2009 for further discussion on this issue).

THEOREM: If $$X_1,...,X_n$$ are IID conditional on $$\theta$$ then they are exchangeable.

PROOF: Choose and arbitrary permutation $$\pi$$ on the set $$\{ 1,...,n \}$$. Since the values $$X_1,...,X_n$$ are IID conditional on $$\theta$$ we have:

\begin{aligned} \mathbb{P}(X_1 \leqslant x_1,...,X_n \leqslant x_n | \theta) &= \prod_{i=1}^n \mathbb{P}(X_i \leqslant x_i | \theta) \\[6pt] &= \prod_{i=1}^n \mathbb{P}(X_{\pi(i)} \leqslant x_i | \theta) \\[6pt] &= \mathbb{P}(X_{\pi(1)} \leqslant x_1,...,X_{\pi(n)} \leqslant x_n | \theta). \\[6pt] \end{aligned}

(The second step above follows simply by taking the product of values in the order of the permutation - it follows from the associativity of multiplication.) We then have:

\begin{aligned} \mathbb{P}(X_1 \leqslant x_1,...,X_n \leqslant x_n) &= \mathbb{E}_\theta \bigg[ \mathbb{P}(X_1 \leqslant x_1,...,X_n \leqslant x_n | \theta) \bigg] \\[6pt] &= \mathbb{E}_\theta \bigg[ \mathbb{P}(X_{\pi(1)} \leqslant x_1,...,X_{\pi(n)} \leqslant x_n | \theta) \bigg] \\[6pt] &= \mathbb{P}(X_{\pi(1)} \leqslant x_1,...,X_{\pi(n)} \leqslant x_n), \\[6pt] \end{aligned}

which establishes exchangeability of $$X_1,...,X_n$$. $$\blacksquare$$

No, I think your reasoning is right. Exchangeability was a very important property to de Finetti in his development of probability theory (which is Bayesian). It also is important regarding permutation tests. Often in doing statistical inference we assume observations are independent and identically distributed and this of course implies exchangeability.

• I am surprised the word "exchangeability" does not come up much more often when talking about Bayesian statistics. maybe I am missing it. – singelton Aug 16 '12 at 17:13
• @singleton I agree with you. It is not mentioned a lot by Bayesians except for followers of deFinetti . Consider for example the work of Diaconis and Freedman. – Michael Chernick Aug 16 '12 at 17:42
• @MichaelChernick I remember a recent discussion between Procrastinaor and you about whether we assume or not a true value of $\theta$ in a Bayesian model. Please do you have the link to this discussion ? This is somewhat related to the present post. – Stéphane Laurent Aug 16 '12 at 20:22
• I am not sure. There are two posts. I think this might be it although procrastinator was not involved in the discussion Fully Bayesian vs Bayesian. The other possibility is Hierarchical Bayes vs Empirical Bayes – Michael Chernick Aug 16 '12 at 22:33