# What is so cool about de Finetti's representation theorem?

From Theory of Statistics by Mark J. Schervish (page 12):

Although DeFinetti's representation theorem 1.49 is central to motivating parametric models, it is not actually used in their implementation.

How is the theorem central to parametric models?

-
(+1) Page number? –  cardinal Aug 16 '12 at 17:45
I think it is central to Bayesian models. I was just discussing this with singleton. It's importance in Bayesian statistics gets overlooked except by those Bayesians who were followers of deFinetti. See this reference of Diaconis and Freedman from 1980 –  Michael Chernick Aug 16 '12 at 17:46
@cardinal: page 12 (I updated the question). –  gui11aume Aug 16 '12 at 18:38
Note that Schervish said "... central to $\textbf{motivating}$ parametric models...". –  Zen Aug 16 '12 at 19:45
I've often wondered how much of the representation is "real" and how much is based on particular interpretations of the theorem. It can be just as easily used for describing a prior distribution as for describing a model. –  probabilityislogic Sep 1 '12 at 23:49

De Finetti's Representation Theorem gives in a single take, within the subjectivistic interpretation of probabilities, the "raison d'être" of statistical models and the meaning of parameters and their prior distributions.

Suppose that the random variables $X_1,\dots,X_n$ represent the results of successive tosses of a coin, with the values $1$ and $0$ corresponding to the results "Heads" and "Tails", respectively. Analysing, within the context of a subjectivistic interpretation of the probability calculus, the meaning of the usual frequentist model under which the $X_i$'s are independent and identically distributed, De Finetti pointed out that the condition of independence would imply, for example, that $$P\{X_n=x_n\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\} = P\{X_n=x_n\} \, ,$$ and, therefore, the results of the first $n-1$ tosses would not change my uncertainty about the result of $n$-th toss. For example, if I believe $\textit{a priori}$ that this is a balanced coin, then, after getting the information that the first $999$ tosses turned out to be "Heads", I would still believe, conditionally on that information, that the probability of getting "Heads" on toss 1000 is equal to $1/2$. Effectively, the hypothesis of independence of the $X_i$'s would imply that it is impossible to learn anything about the coin by observing the results of its tosses.

This situation led De Finetti to search for a condition weaker than independence that would resolve this apparent contradiction. The key for De Finetti's solution was a kind of distributional symmetry known as exchangeability.

$\textbf{Definition.}$ For a given finite set $\{X_i\}_{i=1}^n$ of random objects, let $\mu_{X_1,\dots,X_n}$ denote their joint distribution. This finite set is exchangeable if $\mu_{X_1,\dots,X_n} = \mu_{X_{\pi(1)},\dots,X_{\pi(n)}}$, for every permutation $\pi:\{1,\dots,n\}\to\{1,\dots,n\}$. A sequence $\{X_i\}_{i=1}^\infty$ of random objects is exchangeable if each of its finite subsets are exchangeable.

Supposing only that the sequence of random variables $\{X_i\}_{i=1}^\infty$ is exchangeable, De Finetti proved a notable theorem that sheds light on the meaning of commonly used statistical models. In the particular case when the $X_i$'s take the values $0$ and $1$, De Finetti's Representation Theorem says that $\{X_i\}_{i=1}^\infty$ is exchangeable if and only if there is a random variable $\Theta:\Omega\to[0,1]$, with distribution $\mu_\Theta$, such that $$P\{X_1=x_1,\dots,X_n=x_n\} = \int_{[0,1]} \theta^s(1-\theta)^{n-s}\,d\mu_\Theta(\theta) \, ,$$ where $s=\sum_{i=1}^n x_i$. Moreover, we have that $$\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i \xrightarrow[n\to\infty]{} \Theta \qquad \textrm{almost surely},$$ which is known as De Finetti's Strong Law of Large Numbers.

This Representation Theorem shows how statistical models emerge in a Bayesian context: under the hypothesis of exchangeability of the observables $\{X_i\}_{i=1}^\infty$, $\textbf{there is}$ a $\textit{parameter}$ $\Theta$ such that, given the value of $\Theta$, the observables are $\textit{conditionally}$ independent and identically distributed. Moreover, De Finetti's Strong law shows that our prior opinion about the unobservable $\Theta$, represented by the distribution $\mu_\Theta$, is the opinion about the limit of $\bar{X}_n$, before we have any data. The parameter $\Theta$ plays the role of a useful subsidiary construction, which allows us to obtain conditional probabilities involving only observables through relations like $$P\{X_n=1\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\} = \mathrm{E}\left[\Theta\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\right] \, .$$

-
Thank you for this insightful answer! Your point about independence is a very important one that I realize for the first time. –  gui11aume Aug 16 '12 at 23:14

Everything is mathematically correct in Zen's answer. However I disagree on some points. Please be aware that I don't claim/believe my point of view is the good one; on the contrary I feel these points are not entirely clear for me yet. These are somewhat philosophical questions about which I like to discuss (and a good English exercise for me), and I am also interested in any advice.

• About the example with $999$ "Heads", Zen comment: "the hypothesis of independence of the $X_i$'s would imply that it is impossible to learn anything about the coin by observing the results of its tosses." This is not true from the frequentist perspective: learning about the coin means learning about $\theta$, which is possible by estimating (point-estimate or confidence interval) $\theta$ from the previous $999$ results. If the frequentist observe $999$ "Heads" then he/she concludes that $\theta$ is likely close to $1$, and so is $\Pr(X_n=1)$ consequently.

• By the way, in this coin-tossing example, what is the random $\Theta$ ? Imagining each of two people play a coin-tossing game an infinite number of times with the same coin, why would they find a different $\theta = \bar X_\infty$ ? I have in mind that the characteristic of the coin-tossing is the fixed $\theta$ which is the common value of $\bar X_\infty$ for any gamer ("almost any gamer" for technical mathemathical reasons). A more concrete example for which there's no interpretable random $\Theta$ is the case of a random sampling with replacment in a finite population of $0$ and $1$.

• About Schervish's book and the question raised by the OP I think (quickly speaking) Schervish means that exchangeability is a "cool" assumption and then deFinetti's theorem is "cool" because it says that every exchangeable model has a parametric representation. Of course I totally agree. However if I assume an exchangeable model such as $(X_i\mid\Theta=\theta)\sim_\text{iid} \text{Bernoulli}(\theta)$ and $\Theta \sim \text{Beta}(a,b)$ then I would be interested in performing inference about $a$ and $b$, not about the realization of $\Theta$. If I am only interested in the realization of $\Theta$ then I don't see any interest in assuming exchangeability.

It's late...

-
Hi Stéphane! Thank you for your comments on my answer. About your first point that $\textbf{"this is not true from the frequentist perspective"}$, in my answer everything is stated in a Bayesian context. There is no real attempt to establish a contrast with other inference paradigms. In short, I've tried to express what De Finetti's theorem means for me, as a Bayesian. –  Zen Aug 16 '12 at 23:39
About your second bullet: the random $\Theta$ is (a.s.) the limit of $\bar{X}_n$, as stated in De Finetti's LLN. So, when some Bayesian says that my prior for $\Theta$ is $\mu_\Theta$, he means that this distribution represents his uncertainty about this limit, before having access to data. Different Bayesians may have different priors, but, with suitable regularity conditions, they will have $\textit{a posteriori}$ agreement about $\Theta$ (similar posteriors), as they get more and more information about results of the tosses. –  Zen Aug 16 '12 at 23:51
The fixed but unknown $\theta$ is not a Bayesian concept. –  Zen Aug 16 '12 at 23:58
About your third bullet, given: 1) That Schervish is a Bayesian statistician; 2) The ammount of time and energy he spends discussing exchangeability in his book; I believe that to him the role of De Finetti's theorem is very deep, going well beyond coolness. But I agree that it is very cool, anyway! –  Zen Aug 17 '12 at 0:03
Finally, your last comment about the Bernoulli experiment isn't right about the concepts: for a Bayesian, the $a$ and $b$ are $\textbf{known}$ positive real numbers. It doesn't make sense for a Bayesian, in this specific scenario, to "perform inference about $a$ and $b$". He just updates his prior given the number of observed success. I feel that you understand this, but, as you've said, it's late, and you are probably tired. –  Zen Aug 17 '12 at 0:15