# What is so cool about de Finetti's representation theorem?

From Theory of Statistics by Mark J. Schervish (page 12):

Although DeFinetti's representation theorem 1.49 is central to motivating parametric models, it is not actually used in their implementation.

How is the theorem central to parametric models?

• I think it is central to Bayesian models. I was just discussing this with singleton. It's importance in Bayesian statistics gets overlooked except by those Bayesians who were followers of deFinetti. See this reference of Diaconis and Freedman from 1980 – Michael R. Chernick Aug 16 '12 at 17:46
• @cardinal: page 12 (I updated the question). – gui11aume Aug 16 '12 at 18:38
• Note that Schervish said "... central to $\textbf{motivating}$ parametric models...". – Zen Aug 16 '12 at 19:45
• I've often wondered how much of the representation is "real" and how much is based on particular interpretations of the theorem. It can be just as easily used for describing a prior distribution as for describing a model. – probabilityislogic Sep 1 '12 at 23:49

De Finetti's Representation Theorem gives in a single take, within the subjectivistic interpretation of probabilities, the raison d'être of statistical models and the meaning of parameters and their prior distributions.

Suppose that the random variables $X_1,\dots,X_n$ represent the results of successive tosses of a coin, with values $1$ and $0$ corresponding to the results "Heads" and "Tails", respectively. Analyzing, within the context of a subjectivistic interpretation of the probability calculus, the meaning of the usual frequentist model under which the $X_i$'s are independent and identically distributed, De Finetti observed that the condition of independence would imply, for example, that $$P\{X_n=x_n\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\} = P\{X_n=x_n\} \, ,$$ and, therefore, the results of the first $n-1$ tosses would not change my uncertainty about the result of $n$-th toss. For example, if I believe $\textit{a priori}$ that this is a balanced coin, then, after getting the information that the first $999$ tosses turned out to be "Heads", I would still believe, conditionally on that information, that the probability of getting "Heads" on toss 1000 is equal to $1/2$. Effectively, the hypothesis of independence of the $X_i$'s would imply that it is impossible to learn anything about the coin by observing the results of its tosses.

This observation led De Finetti to the introduction of a condition weaker than independence that resolves this apparent contradiction. The key to De Finetti's solution is a kind of distributional symmetry known as exchangeability.

$\textbf{Definition.}$ For a given finite set $\{X_i\}_{i=1}^n$ of random objects, let $\mu_{X_1,\dots,X_n}$ denote their joint distribution. This finite set is exchangeable if $\mu_{X_1,\dots,X_n} = \mu_{X_{\pi(1)},\dots,X_{\pi(n)}}$, for every permutation $\pi:\{1,\dots,n\}\to\{1,\dots,n\}$. A sequence $\{X_i\}_{i=1}^\infty$ of random objects is exchangeable if each of its finite subsets are exchangeable.

Supposing only that the sequence of random variables $\{X_i\}_{i=1}^\infty$ is exchangeable, De Finetti proved a notable theorem that sheds light on the meaning of commonly used statistical models. In the particular case when the $X_i$'s take the values $0$ and $1$, De Finetti's Representation Theorem says that $\{X_i\}_{i=1}^\infty$ is exchangeable if and only if there is a random variable $\Theta:\Omega\to[0,1]$, with distribution $\mu_\Theta$, such that $$P\{X_1=x_1,\dots,X_n=x_n\} = \int_{[0,1]} \theta^s(1-\theta)^{n-s}\,d\mu_\Theta(\theta) \, ,$$ in which $s=\sum_{i=1}^n x_i$. Moreover, we have that $$\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i \xrightarrow[n\to\infty]{} \Theta \qquad \textrm{almost surely},$$ which is known as De Finetti's Strong Law of Large Numbers.

This Representation Theorem shows how statistical models emerge in a Bayesian context: under the hypothesis of exchangeability of the observables $\{X_i\}_{i=1}^\infty$, $\textbf{there is}$ a $\textit{parameter}$ $\Theta$ such that, given the value of $\Theta$, the observables are $\textit{conditionally}$ independent and identically distributed. Moreover, De Finetti's Strong law shows that our prior opinion about the unobservable $\Theta$, represented by the distribution $\mu_\Theta$, is the opinion about the limit of $\bar{X}_n$, before we have information about the values of the realizations of any of the $X_i$'s. The parameter $\Theta$ plays the role of a useful subsidiary construction, which allows us to obtain conditional probabilities involving only observables through relations like $$P\{X_n=1\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\} = \mathrm{E}\left[\Theta\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\right] \, .$$

• Thank you for this insightful answer! Your point about independence is a very important one that I realize for the first time. – gui11aume Aug 16 '12 at 23:14
• ("a useful" was better :) ) – Neil G Jan 4 '16 at 17:54
• I'm having a hard time understanding the statement "there exists parameter $\Theta$ so that (given $\Theta$) $X_i$ are iid." From the representation theorem, it seems that all we can derive is that $E [\theta^s (1-\theta)^s] = E[P(X_i = x_i \, \forall \, i | \theta) ]$. That is, the expected value of the true density is the same as the expected value of the iid bernoulli density with parameter $\theta$. Could you clarify for me how we can drop the expected value so that we make a claim about the true density itself? – user795305 Jul 26 '17 at 19:52
• The integrand is $\Pr\{X_1=x_1,\dots,X_n=x_n\mid\Theta=\theta\}$. Since it factors as $\prod_{i=1}^n \Pr\{X_i=x_i\mid\Theta=\theta\}=\prod_{i=1}^n \theta^{x_i}(1-\theta)^{1-x_i}$, the $X_i$'s are conditionally iid given $\Theta=\theta$. – Zen Jul 27 '17 at 0:36
• @Zen Thanks! I understand the first sentence, however the part "since it factors as $\prod_{i=1}^n \Pr\{X_i=x_i\mid\Theta=\theta\}=\prod_{i=1}^n \theta^{x_i}(1-\theta)^{1-x_i}$" is still unclear to me. How do you know it factors that way? It seems like you're dropping the expected value from the identity I wrote in my previous comment, but I'm not sure how that's justified. – user795305 Jul 28 '17 at 2:49

Everything is mathematically correct in Zen's answer. However I disagree on some points. Please be aware that I don't claim/believe my point of view is the good one; on the contrary I feel these points are not entirely clear for me yet. These are somewhat philosophical questions about which I like to discuss (and a good English exercise for me), and I am also interested in any advice.

• About the example with $999$ "Heads", Zen comment: "the hypothesis of independence of the $X_i$'s would imply that it is impossible to learn anything about the coin by observing the results of its tosses." This is not true from the frequentist perspective: learning about the coin means learning about $\theta$, which is possible by estimating (point-estimate or confidence interval) $\theta$ from the previous $999$ results. If the frequentist observe $999$ "Heads" then he/she concludes that $\theta$ is likely close to $1$, and so is $\Pr(X_n=1)$ consequently.

• By the way, in this coin-tossing example, what is the random $\Theta$ ? Imagining each of two people play a coin-tossing game an infinite number of times with the same coin, why would they find a different $\theta = \bar X_\infty$ ? I have in mind that the characteristic of the coin-tossing is the fixed $\theta$ which is the common value of $\bar X_\infty$ for any gamer ("almost any gamer" for technical mathemathical reasons). A more concrete example for which there's no interpretable random $\Theta$ is the case of a random sampling with replacment in a finite population of $0$ and $1$.

• About Schervish's book and the question raised by the OP I think (quickly speaking) Schervish means that exchangeability is a "cool" assumption and then deFinetti's theorem is "cool" because it says that every exchangeable model has a parametric representation. Of course I totally agree. However if I assume an exchangeable model such as $(X_i\mid\Theta=\theta)\sim_\text{iid} \text{Bernoulli}(\theta)$ and $\Theta \sim \text{Beta}(a,b)$ then I would be interested in performing inference about $a$ and $b$, not about the realization of $\Theta$. If I am only interested in the realization of $\Theta$ then I don't see any interest in assuming exchangeability.

It's late...

• Hi Stéphane! Thank you for your comments on my answer. About your first point that $\textbf{"this is not true from the frequentist perspective"}$, in my answer everything is stated in a Bayesian context. There is no real attempt to establish a contrast with other inference paradigms. In short, I've tried to express what De Finetti's theorem means for me, as a Bayesian. – Zen Aug 16 '12 at 23:39
• About your second bullet: the random $\Theta$ is (a.s.) the limit of $\bar{X}_n$, as stated in De Finetti's LLN. So, when some Bayesian says that my prior for $\Theta$ is $\mu_\Theta$, he means that this distribution represents his uncertainty about this limit, before having access to data. Different Bayesians may have different priors, but, with suitable regularity conditions, they will have $\textit{a posteriori}$ agreement about $\Theta$ (similar posteriors), as they get more and more information about results of the tosses. – Zen Aug 16 '12 at 23:51
• The fixed but unknown $\theta$ is not a Bayesian concept. – Zen Aug 16 '12 at 23:58
• About your third bullet, given: 1) That Schervish is a Bayesian statistician; 2) The ammount of time and energy he spends discussing exchangeability in his book; I believe that to him the role of De Finetti's theorem is very deep, going well beyond coolness. But I agree that it is very cool, anyway! – Zen Aug 17 '12 at 0:03
• To clarify my point of view: I don't believe there is a random $\theta$ in a "basic" (non hierarchical) Bayesian model. There is a fixed unknown $\theta$, and the prior distribution describes the belief about it. The role of the random variable $\Theta$ is just the mathematical treatment of the Bayesian inference, it does not have any interpretation in the experiment. If you really assume exchangeable but not independent observations, such as the example of my third bullet, then you have to put hyperpriors on $a$ and $b$. – Stéphane Laurent Aug 17 '12 at 9:07

You guys might be interested in a paper on this subject (journal subscription required for access - try accessing it from your university):

O'Neill, B. (2011) Exchangeability, correlation and Bayes' Effect. International Statistical Review 77(2), pp. 241-250.

This paper discusses the representation theorem as the basis for both Bayesian and frequentist IID models, and also applies it to a coin-tossing example. It should clear up the discussion of the assumptions of the frequentist paradigm. It actually uses a broader extension to the representation theorem going beyond the binomial model, but it should still be useful.

• Is there maybe a working paper version of this you have? I don't have access atm :-( – IMA Jun 9 '13 at 20:46
• @Stats I've read that paper after see your answer. I have to say, that is the best paper illustrating Bayesian and Frequentist on that issue I have ever seen. I wish I would have read this paper much earlier. (+1) – KevinKim May 28 '17 at 13:18