The definition of an exchangeabilty for a finite sequence says that, if we have random variables $X_1,\ldots,X_n$, then for each permutation $\pi: \{1,\ldots,n\}\rightarrow\{1,\ldots,n\}$, the joint distribution does not change. This can be stated in terms of distribution function as: $$ F_{X_1,\ldots,X_n}(x_1,\ldots,x_n)=F_{X_{\pi(1)},\ldots,X_{\pi(n)}}(x_{\pi(1)},\ldots,x_{\pi(n)}) $$ A way in which i understand this concept is that random variables in a set are "similar", and each of them can "act" as another one, with no impact of uncertainty about values of whole vector. In other words, the index set provides no added value. However, i often meet an example of exchangeability as follows: suppose we have a set of 5 binary random variables, which describe an experiment of sequence drawing 5 balls without replacement from an urn, which contains 2 blue and 3 red balls: $$ X_i(\omega)=\mathbb{1_{\{\mathbf{red}\}}}(\omega) $$ Then exchangeability is shown in terms of equality of probability of every 5-digit sequence, which for me is an autonomic statement about joint distribution of random vector $(X_1,\ldots,X_5)$. Could someone highlight me a connection between these two representations and show, how they arise from each other?
-
1$\begingroup$ Not sure if I fully understand your question. But note that drawing from an urn without replacement does not automatically imply exchangeability. For example, if you know that red balls have been poured into an urn, and then blue balls, and no shaking or mixing has been done afterwards, and you also know that the drawing will most likely happen from the surface layers, then your probabilities for sequences in which blue draws come first and red later are higher than for sequences with red first and blue later. Your distribution for sequences is therefore not exchangeable. $\endgroup$– pglpmJul 6, 2020 at 7:28
-
$\begingroup$ For the latter case see the concrete study in Jaynes's book, chapter 3, "Correction for correlations". $\endgroup$– pglpmJul 6, 2020 at 7:29
2 Answers
Instead of focusing only on the distribution function, let's focus on equality in distribution.
A finite sequence of random variables $X_1, \ldots, X_n$ is exchangeable if for every permutation $\pi$ we have $$ X_1, \ldots, X_N =_d X_{\pi(1)}, \ldots, X_{\pi(n)} $$ where $=_d$ means equality in distribution.
Equality in distribution is equivalent to equality of the distribution function, equality of the probability distribution function if the random variables are continuous and equality of the probability mass function if the random variables are discrete.
Now let's get back to your example, suppose that we have a realization $X_1, \ldots, X_5 = 1, 1, 1, 0, 0$. We can compute the joint law of $X_1, \ldots, X_n$ by hand an in particular we have $$ P(X_1, \ldots, X_5 = 1, 1, 1, 0, 0) = \frac{3}{5} \frac{2}{4} \frac{1}{3} \frac{1}{2} \frac{1}{1} $$ Looking at the expression above, it is clear that a permutation of the indices will leave the denominators unchanged and only the numerators will permute, leaving overall the joint probability unchanged, which is the definition of exchangeability
-
$\begingroup$ I need one clarification: does permutation of numerators (implied by permutation of indices) spring from the formula of probability of intersection? For example, for "inverse" vector $(X_5,\ldots,X_1)=(0,0,1,1,1)$ we have, that $\mathbb{P}(\{X_5=0\}\cap\ldots\cap\{X_1=1\})=\mathbb{P}(\{X_5=0\})\times\mathbb{P}(\{X_4=0\}\mid \{X_5=0\})\times\ldots\times\mathbb{P}(\{X_1=1\}\mid\{X_5=0\}\cap\ldots\cap\{X_2=1\})$. Am i right here? $\endgroup$ Jul 6, 2020 at 12:03
-
1$\begingroup$ Yes totally right, you just keep conditioning on the past draws $\endgroup$– mariob6Jul 6, 2020 at 12:22
Something that might be helpful in unifying these two views is the Hewitt-Savage(-de Finetti) representation theorem.
The theorem says that $X_1,\dots,X_n$ are exchangeable precisely when they are independent and identically distributed conditional on some additional information. This is important in Bayesian statistics, because it means that an exchangeable sequence (which seems a reasonable thing to assume) can be modelled as an iid sequence plus a prior (which is convenient mathematically).
For binary variables the extra information is just the probability. If $P$ is a random variable between 0 and 1, and $X_i|P\sim \mathrm{Bern}(P)$, then $X_i$ are exchangeable, and they are conditionally iid given $P$. That's the de Finetti result.
Hewitt and Savage showed this was true generally, not just for binary sequences: a sequence is exchangeable if and only if it's iid conditional on some extra information, in this case the 'tail $\sigma$-field'
