What is so cool about de Finetti's representation theorem?

Question

From Theory of Statistics by Mark J. Schervish (page 12):

Although DeFinetti's representation theorem 1.49 is central to motivating parametric models, it is not actually used in their implementation.

How is the theorem central to parametric models?

Answer 1

De Finetti's Representation Theorem gives in a single take, within the subjectivistic interpretation of probabilities, the "raison d'être" of statistical models and the meaning of parameters and their prior distributions.

Suppose that the random variables $X_1,\dots,X_n$ represent the results of successive tosses of a coin, with the values $1$ and $0$ corresponding to the results "Heads" and "Tails", respectively. Analysing, within the context of a subjectivistic interpretation of the probability calculus, the meaning of the usual frequentist model under which the $X_i$'s are independent and identically distributed, De Finetti pointed out that the condition of independence would imply, for example, that $$ P\{X_n=x_n\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\} = P\{X_n=x_n\} \, , $$ and, therefore, the results of the first $n-1$ tosses would not change my uncertainty about the result of $n$-th toss. For example, if I believe $\textit{a priori}$ that this is a balanced coin, then, after getting the information that the first $999$ tosses turned out to be "Heads", I would still believe, conditionally on that information, that the probability of getting "Heads" on toss 1000 is equal to $1/2$. Effectively, the hypothesis of independence of the $X_i$'s would imply that it is impossible to learn anything about the coin by observing the results of its tosses.

This situation led De Finetti to search for a condition weaker than independence that would resolve this apparent contradiction. The key for De Finetti's solution was a kind of distributional symmetry known as exchangeability.

$\textbf{Definition.}$ For a given finite set $\{X_i\}_{i=1}^n$ of random objects, let $\mu_{X_1,\dots,X_n}$ denote their joint distribution. This finite set is exchangeable if $\mu_{X_1,\dots,X_n} = \mu_{X_{\pi(1)},\dots,X_{\pi(n)}}$, for every permutation $\pi:\{1,\dots,n\}\to\{1,\dots,n\}$. A sequence $\{X_i\}_{i=1}^\infty$ of random objects is exchangeable if each of its finite subsets are exchangeable.

Supposing only that the sequence of random variables $\{X_i\}_{i=1}^\infty$ is exchangeable, De Finetti proved a notable theorem that sheds light on the meaning of commonly used statistical models. In the particular case when the $X_i$'s take the values $0$ and $1$, De Finetti's Representation Theorem says that $\{X_i\}_{i=1}^\infty$ is exchangeable if and only if there is a random variable $\Theta:\Omega\to[0,1]$, with distribution $\mu_\Theta$, such that $$ P\{X_1=x_1,\dots,X_n=x_n\} = \int_{[0,1]} \theta^s(1-\theta)^{n-s}\,d\mu_\Theta(\theta) \, , $$ where $s=\sum_{i=1}^n x_i$. Moreover, we have that $$ \bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i \xrightarrow[n\to\infty]{} \Theta \qquad \textrm{almost surely}, $$ which is known as De Finetti's Strong Law of Large Numbers.

This Representation Theorem shows how statistical models emerge in a Bayesian context: under the hypothesis of exchangeability of the observables $\{X_i\}_{i=1}^\infty$, $\textbf{there is}$ a $\textit{parameter}$ $\Theta$ such that, given the value of $\Theta$, the observables are $\textit{conditionally}$ independent and identically distributed. Moreover, De Finetti's Strong law shows that our prior opinion about the unobservable $\Theta$, represented by the distribution $\mu_\Theta$, is the opinion about the limit of $\bar{X}_n$, before we have any data. The parameter $\Theta$ plays the role of a useful subsidiary construction, which allows us to obtain conditional probabilities involving only observables through relations like $$ P\{X_n=1\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\} =E\left[\Theta\mid X_1=x_1,\dots,X_{n-1}=x_{n-1}\right] \, . $$

Answer 2

Everything is mathematically correct in Zen's answer. However I disagree on some points. Please be aware that I don't claim/believe my point of view is the good one; on the contrary I feel these points are not entirely clear for me yet. These are somewhat philosophical questions about which I like to discuss (and a good English exercise for me), and I am also interested in any advice.

• About the example with $999$ "Heads", Zen comment: "the hypothesis of independence of the $X_i$'s would imply that it is impossible to learn anything about the coin by observing the results of its tosses." This is not true from the frequentist perspective: learning about the coin means learning about $\theta$, which is possible by estimating (point-estimate or confidence interval) $\theta$ from the previous $999$ results. If the frequentist observe $999$ "Heads" then he/she concludes that $\theta$ is likely close to $1$, and so is $\Pr(X_n=1)$ consequently.

• By the way, in this coin-tossing example, what is the random $\Theta$ ? Imagining each of two people play a coin-tossing game an infinite number of times with the same coin, why would they find a different $\theta = \bar X_\infty$ ? I have in mind that the characteristic of the coin-tossing is the fixed $\theta$ which is the common value of $\bar X_\infty$ for any gamer ("almost any gamer" for technical mathemathical reasons). A more concrete example for which there's no interpretable random $\Theta$ is the case of a random sampling with replacment in a finite population of $0$ and $1$.

• About Schervish's book and the question raised by the OP I think (quickly speaking) Schervish means that exchangeability is a "cool" assumption and then deFinetti's theorem is "cool" because it says that every exchangeable model has a parametric representation. Of course I totally agree. However if I assume an exchangeable model such as $(X_i\mid\Theta=\theta)\sim_\text{iid} \text{Bernoulli}(\theta)$ and $\Theta \sim \text{Beta}(a,b)$ then I would be interested in performing inference about $a$ and $b$, not about the realization of $\Theta$. If I am only interested in the realization of $\Theta$ then I don't see any interest in assuming exchangeability.

It's late...

Answer 3

You guys might be interested in a paper on this subject (journal subscription required for access - try accessing it from your university):

O'Neill, B. (2011) Exchangeability, correlation and Bayes' Effect. International Statistical Review 77(2), pp. 241-250.

This paper discusses the representation theorem as the basis for both Bayesian and frequentist IID models, and also applies it to a coin-tossing example. It should clear up the discussion of the assumptions of the frequentist paradigm. It actually uses a broader extension to the representation theorem going beyond the binomial model, but it should still be useful.

Answer 4

I think it is central to Bayesian models. I was just discussing this with singleton. It's importance in Bayesian statistics gets overlooked except by those Bayesians who were followers of deFinetti. See this reference of Diaconis and Freedman from 1980