# Maximizing Mutual Information

Consider random variables $X_i \sim Bern(0.5)$. We know that the conditional entropy $H(X_1,X_2)$ is maximized when $X_2$ is independent of $X_1$, i.e, the joint distribution $\mathbb{P}(X_1=x_1, X_2=x_2)=\mathbb{P}(X_1=x_1)\mathbb{P}(X_2=x_2)$.

Let $Z =\textit{Unordered}\{X_1,X_2,\cdots,X_n\}$, an unordered set of $X_i$'s.

Question: What joint distribution $p(x_1,x_2,\cdots,x_n)$ maximizes

$$H(X_1,X_2)-H(X_1,X_2 | Z )$$

• Note: $I(X_1, X_2;Z) = H(X_1,X_2) - H(X_1, X_2|Z)$. – Vivek Bagaria Mar 9 '17 at 22:37