41
$\begingroup$

I see the concept of 'exchangeability' being used in different contexts (e.g., bayesian models) but I have never understood the term very well.

  1. What does this concept mean?

  2. Under what circumstances is this concept invoked and why?

$\endgroup$
44
$\begingroup$

Exchangeability is meant to capture symmetry in a problem, symmetry in a sense that does not require independence. Formally, a sequence is exchangeable if its joint probability distribution is a symmetric function of its $n$ arguments. Intuitively it means we can swap around, or reorder, variables in the sequence without changing their joint distribution. For example, every IID (independent, identically distributed) sequence is exchangeable - but not the other way around. Every exchangeable sequence is identically distributed, though.

Imagine a table with a bunch of urns on top, each containing different proportions of red and green balls. We choose an urn at random (according to some prior distribution), and then take a sample (without replacement) from the selected urn.

Note that the reds and greens that we observe are NOT independent. And it is maybe not a surprise to learn that the sequence of reds and greens we observe is an exchangeable sequence. What is maybe surprising is that EVERY exchangeable sequence can be imagined this way, for a suitable choice of urns and prior distribution. (see Diaconis/Freedman (1980) "Finite Exchangeable Sequences", Ann. Prob.).

The concept is invoked in all sorts of places, and it is especially useful in Bayesian contexts because in those settings we have a prior distribution (our knowledge of the distribution of urns on the table) and we have a likelihood running around (a model which loosely represents the sampling procedure from a given, fixed, urn). We observe the sequence of reds and greens (the data) and use that information to update our beliefs about the particular urn in our hand (i.e., our posterior), or more generally, the urns on the table.

Exchangeable random variables are especially wonderful because if we have infinitely many of them then we have tomes of mathematical machinery at our fingertips not the least of which being de Finetti's Theorem; see Wikipedia for an introduction.

$\endgroup$
  • 10
    $\begingroup$ (+1) The exchangeability assumption is also at the heart of permutation tests. $\endgroup$ – chl Oct 12 '10 at 15:53
  • 9
    $\begingroup$ Given the question of the when & why of exchangeability, chl's pointer to permutation tests may merit a few additional words. Permutation tests are a nonparametric technique used when normality and similar assumptions are untenable - instead one uses the much weaker "null assumption" of exchangeability, approximates the distribution of a test statistic under this null assumption (by permuting) and looks whether the actually observed test statistic is extreme compared to this null distribution. There is an accessible book by P. Good, "Permutation, Parametric, and Bootstrap Tests of Hypotheses". $\endgroup$ – Stephan Kolassa Oct 12 '10 at 20:43
  • $\begingroup$ @Stephan I like this book! Still, exchangeability is weaker than independence... $\endgroup$ – chl Oct 12 '10 at 21:01
  • $\begingroup$ Thanks very much for this helpful answer, GJK, and for the notes about permutation tests, @StephanKolassa and chi. I'm confused about what role the multiple urns are playing in the example, though. The sequence of reds and greens would be exchangeable with a single, urn, right? What does the possibility of multiple urns (multiple distributions) add? $\endgroup$ – Mars Jul 29 '15 at 16:31
  • 3
    $\begingroup$ I would not trust anything in that book by Good. I once spotted what I though was an error in that book, wrote him, and received a rather vulgar noanswer. $\endgroup$ – kjetil b halvorsen Feb 20 '18 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.