The total correlation (aka multivariate mutual information) between a bunch of random variables $X_1,...,X_n \in \mathcal{X}_1 \times ... \times \mathcal{X}_n$ with joint density $p(x_1,...,x_n)$ and marginals $p_1(x_1),...p_n(x_n)$ is defined as $$ I(X_1,...,X_n) = \int \log\left( \frac{p(x_1,...,x_n)}{p_1(x_1)...p_n(x_n)}\right) p(x_1,...,x_n) dxdy, $$ and is usually seen as a measure of dependence between $X_1,...,X_n$.

In general, $I(X_1,...,X_n) \geq 0$. In the particular case when $X_1,...,X_n$ are independent, $I(X_1,...,X_n) = 0$.

My question is: what happens is we have a weaker assumption than independence? For example, a weaker notion of independence is exchangeability: $$\forall \sigma \in S_n, \forall x_1,...,x_n \in \mathcal{X}_1 \times ... \times \mathcal{X}_n, \; p(x_1,...,x_n) = p(x_{\sigma(1)},...,x_{\sigma(n)}),$$ where $S_n$ is the set of all possible permutations of $\{1,...,n\}$.

Can we say something about $I(X_1,...,X_n)$ when $X_1,...,X_n$ are exchangeable? For example, can we get an upper bound on $I(X_1,...,X_n)$?

  • $\begingroup$ If exchangeability were to hold, I don't think $I(\mathbf{X})$ suddenly attains an upper bound just because of this property. What about exchangeability made you think that it would provide an upper bound if it were to be a property of mutual information? Exchangeability and an upper bound are two separate questions. Outside of the framework of your question, in general, mutual information usually acts as a lower bound for other information theoretic measures $\endgroup$ – develarist Nov 23 '20 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.