I see the concept of 'exchangeability' being used in different contexts (e.g., bayesian models) but I have never understood the term very well.
What does this concept mean?
Under what circumstances is this concept invoked and why?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
11
Exchangeability is meant to capture symmetry in a problem, symmetry in a sense that does not require independence. Formally, a sequence is exchangeable if its joint probability distribution is a symmetric function of its $n$ arguments. Intuitively it means we can swap around, or reorder, variables in the sequence without changing their joint distribution. For example, every IID (independent, identically distributed) sequence is exchangeable - but not the other way around. Every exchangeable sequence is identically distributed, though.
Imagine a table with a bunch of urns on top, each containing different proportions of red and green balls. We choose an urn at random (according to some prior distribution), and then take a sample (without replacement) from the selected urn.
Note that the reds and greens that we observe are NOT independent. And it is maybe not a surprise to learn that the sequence of reds and greens we observe is an exchangeable sequence. What is maybe surprising is that EVERY exchangeable sequence can be imagined this way, for a suitable choice of urns and prior distribution. (see Diaconis/Freedman (1980) "Finite Exchangeable Sequences", Ann. Prob.).
The concept is invoked in all sorts of places, and it is especially useful in Bayesian contexts because in those settings we have a prior distribution (our knowledge of the distribution of urns on the table) and we have a likelihood running around (a model which loosely represents the sampling procedure from a given, fixed, urn). We observe the sequence of reds and greens (the data) and use that information to update our beliefs about the particular urn in our hand (i.e., our posterior), or more generally, the urns on the table.
Exchangeable random variables are especially wonderful because if we have infinitely many of them then we have tomes of mathematical machinery at our fingertips not the least of which being de Finetti's Theorem; see Wikipedia for an introduction.
-
14$\begingroup$ (+1) The exchangeability assumption is also at the heart of permutation tests. $\endgroup$– chlCommented Oct 12, 2010 at 15:53
-
13$\begingroup$ Given the question of the when & why of exchangeability, chl's pointer to permutation tests may merit a few additional words. Permutation tests are a nonparametric technique used when normality and similar assumptions are untenable - instead one uses the much weaker "null assumption" of exchangeability, approximates the distribution of a test statistic under this null assumption (by permuting) and looks whether the actually observed test statistic is extreme compared to this null distribution. There is an accessible book by P. Good, "Permutation, Parametric, and Bootstrap Tests of Hypotheses". $\endgroup$ Commented Oct 12, 2010 at 20:43
-
$\begingroup$ @Stephan I like this book! Still, exchangeability is weaker than independence... $\endgroup$– chlCommented Oct 12, 2010 at 21:01
-
6$\begingroup$ I would not trust anything in that book by Good. I once spotted what I though was an error in that book, wrote him, and received a rather vulgar noanswer. $\endgroup$ Commented Feb 20, 2018 at 14:08
-
1$\begingroup$ @JoshO'Brien There is no contradiction. The point is that "identically distributed" is weaker than "independent, identically distributed", and it is indeed true that every exchangeable sequence of random variables is identically distributed. $\endgroup$ Commented Feb 21, 2023 at 8:03