Why the exchangeability of random variables is essential in hierarchical bayesian models?

Question

Why the exchangeability of random variables is essential for the hierarchical Bayesian modeling?

Answer 1

Exchangeability is not an essential feature of a hierarchical model (at least not at the observational level). It is basically a Bayesian analogue of "independent and identically distributed" from the standard literature. It is simply a way of describing what you know about the situation at hand. This is namely that "shuffling" does not alter your problem. One way I like to think of this is to consider the case where you were given $x_{j}=5$ but you were not told the value of $j$. If learning that $x_{j}=5$ would lead you to suspect particular values of $j$ more than others, then the sequence is not exchangeable. If it tells you nothing about $j$, then the sequence is exchangeable. Note that exhcangeability is "in the information" rather than "in reality" - it depends on what you know.

While exchangeability is not essential in terms of the observed variables, it would probably be quite difficult to fit any model without some notion of exchangeability, because without exchangeability you basically have no justification for pooling observations together. So my guess is that your inferences will be much weaker if you don't have exchangeability somewhere in the model. For example, consider $x_{i}\sim N(\mu_{i},\sigma_{i})$ for $i=1,\dots,N$. If $x_{i}$ are completely exchangeable then this means $\mu_{i}=\mu$ and $\sigma_{i}=\sigma$. If $x_{i}$ are conditionally exchangeable given $\mu_{i}$ then this means $\sigma_{i}=\sigma$. If $x_{i}$ are conditionally exchangeable given $\sigma_{i}$ then this means $\mu_{i}=\mu$. But note that in either of these two "conditionally exchangeable" cases, the quality of inference is reduced compared to the first, because there are an extra $N$ parameters that get introduced into the problem. If we have no exchangeability, then we basically have $N$ unrelated problems.

Basically exchangeability means we can make the inference $x_{i}\to \text{parameters}\to x_{j}$ for any $i$ and $j$ which are partly exchangeable

Answer 2

"Essential" is too vague. But surpressing the technicalities, if the sequence $X=\{X_i\}$ is exchangeable then the $X_i$ are conditionally independent given some unobserved parameter(s) $\Theta$ with a probability distribution $\pi$. That is, $p(X) = \int p(X_i|\Theta)d\pi(\Theta)$. $\Theta$ needn't be univariate or even finite dimensional and may be further represented as a mixture, etc.

Exchangability is essential in the sense that these conditional independence relationships allow us to fit models we almost certainly couldn't otherwise.

Answer 3

It isn't! I'm no expert here, but i'll give my two cents. In general when you have a hierarchical model, say

$y|\Theta_{1} \sim \text{N}(X\Theta_{1},\sigma^2)$

$\Theta_{1}|\Theta_{2} \sim\text{N}(W\Theta_{2},\sigma^2)$

We make conditional independence assumptions, i.e., conditional on $\Theta_{2}$, the $\Theta_{1}$ are exchangeable. If the second level is not exchangeable, than you can incluce another level that makes it exchangeable. But even in the case that you can't make an assumption of exchaganbelity, the model may still be a good fit to your data at the first level.

Last, but not least, exchangeability is important only if you wanna think in terms of De Finetti's representation theorem. You might just think that priors are regularization tools that help you to fit your model. In this case, the exchangeability assumption is as good as it is your model fit to the data. In other words, if you think of Bayesian hierarchical model as way to get abetter fit to your data, then exchangeability is not essential in any sense.