I recently read a chapter about Subjectivist notions of Probability in Gillies (2012) and I stumbled upon the concept of exchangeability as the Bayesian version of independence (or something similar at least). No I understand that stochastic independence prohibits me from learning from experience when I repeat an experiment, as if my belief of the result on the n-th trial is independent of the previous results my posterior beliefs will not change. My question is now whether stochastic indepedence does play a role in Bayesian statistics or whether it is not an important concept at all.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The relationship between exchangeability and independence for Bayesian analysis is discussed in detail in O'Neill, B. (2009) Exchangeability, Correlation and Bayes' Effect. International Statistical Review 77(2), pp. 241 - 250. This paper contrasts Bayesian and classical models built on IID data and discusses statistical dependence in each of these cases.
Marginal (i.e., unconditional) independence is a very strong property - if events are statistically independent then (by definition) we cannot learn about one from observing the other. For this reason, statistical models generally involve assumptions of conditional independence, given some underlying distribution or parameters. The exact conceptual framework depends on whether one is using Bayesian methods or classical methods. The former involves explicit dependence between observable values, while the latter involves a (complicated and subtle) implicit form of dependence. Understanding this issue properly requires a bit of understanding of classical versus Bayesian statistics.
Statistical models will often say they use an assumption that sequences of random variables are "independent and identically distributed (IID)". For example, you might have an observable sequence $X_1, X_2, X_3, ... \sim \text{IID N} (\mu, \sigma^2)$, which means that each observable random variable $X_i$ is normally distributed with mean $\mu$ and standard deviation $\sigma$. Each of the random variables in the sequence is "independent" of the others in the sense that its outcome does not change the stated distribution of the other values, which is conditional on these parameters. In this kind of model we use the observed values of the sequence to estimate the parameters in the model, and we can then in turn predict unobserved values of the sequence. This necessarily involves using some observed values to learn about others.
Bayesian statistics: Everything is conceptually simple. The sequence $X_1, X_2, X_3, ... $ is assumed to be exchangeable, which implies that its elements are conditionally IID given the parameters $\mu$ and $\sigma$. We treat those unknown parameters as random variables, and this allows us to derive the predictive distribution, which measures the unconditional dependence between the observable values. Given any non-degenerate prior distribution for these parameters, the values in the observable sequence are (unconditionally) dependent, generally with positive correlation. Hence, it makes perfect sense that we use observed outcomes to predict later unobserved outcomes - they are conditionally independent, but unconditionally dependent.
Classical statistics: This is quite complicated and subtle. Assume $X_1, X_2, X_3, ... $ is IID given the parameters $\mu$ and $\sigma$, but treat those parameters as "unknown constants". Since the parameters are treated as constants, there is no clear difference between conditional and unconditional independence in this case. Nevertheless, we still use the observed values to estimate the parameters and make predictions of the unobserved values. Hence, we use the observed outcomes to predict later unobserved outcomes even though they are notionally "independent" of each other. This apparent incongruity is discussed in detail in the above-cited paper.