Problem Setting

Let $X_1, X_2,\cdots,X_m$ be identical and marginally $Bern(p=0.5)$ random variables. There is no restriction on the joint distribution of $X_1, X_2,\cdots,X_m$.


The entropy $H(X_1, X_2,\cdots,X_m)$ is maximized (over all possible joint distributions) when $X_i's$ are independent. This can be proved by expanding the entropy term using chain rule


Is the entropy of their sum, $S=X_1+X_2+\cdots+X_m$ also maximized when they are independent?

  • $\begingroup$ Are we seeking to maximize the entropy (of the sum) over all possible joint distributions of $X_1,\ldots,X_m$ with the condition that the marginals need to be $Bern(0.5)$? $\endgroup$ – Juho Kokkala Aug 5 '16 at 9:43
  • $\begingroup$ @Juho: Yes. I'll update the question. $\endgroup$ – Vivek Bagaria Aug 5 '16 at 17:55

No. The entropy of $S = X_1 + X_2+\cdots+X_m$ is maximized when $S$ is a uniformly distributed in $[m].$

Example for $m = 2$

$$ P(X_1 = 0, X_2 = 0 ) = 1/3 \\ P(X_1 = 1, X_2 = 0 ) = 1/6 \\ P(X_1 = 0, X_2 = 1 ) = 1/6 \\ P(X_1 = 1, X_2 = 1 ) = 1/3 $$ Tha above joint distribution in $X_1$, $X_2$ has the highest entropy for $H(X_1 +X_2)$. Also, by symmetry, $X_1$, $X_2$ are indentically distributed in the above distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.