Maximizing entropy for sum of random variables - Cross Validated most recent 30 from stats.stackexchange.com 2019-07-21T15:28:22Z https://stats.stackexchange.com/feeds/question/228400 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/228400 1 Maximizing entropy for sum of random variables Vivek Bagaria https://stats.stackexchange.com/users/99046 2016-08-05T08:47:42Z 2016-08-24T23:56:17Z <p><strong>Problem Setting</strong></p> <p>Let \$X_1, X_2,\cdots,X_m\$ be identical and marginally \$Bern(p=0.5)\$ random variables. There is no restriction on the joint distribution of \$X_1, X_2,\cdots,X_m\$.</p> <p><strong>Observation</strong></p> <p>The entropy \$H(X_1, X_2,\cdots,X_m)\$ is maximized (over all possible joint distributions) when \$X_i's\$ are independent. This can be proved by expanding the entropy term using <a href="https://en.wikipedia.org/wiki/Conditional_entropy#Chain_rule" rel="nofollow">chain rule</a></p> <p><strong>Question</strong></p> <p>Is the entropy of their sum, \$S=X_1+X_2+\cdots+X_m\$ also maximized when they are independent? </p> https://stats.stackexchange.com/questions/228400/-/231573#231573 0 Answer by Vivek Bagaria for Maximizing entropy for sum of random variables Vivek Bagaria https://stats.stackexchange.com/users/99046 2016-08-24T23:56:17Z 2016-08-24T23:56:17Z <p><strong>No</strong>. The entropy of \$S = X_1 + X_2+\cdots+X_m\$ is maximized when \$S\$ is a uniformly distributed in \$[m].\$</p> <p>Example for \$m = 2\$ </p> <p>\$\$ P(X_1 = 0, X_2 = 0 ) = 1/3 \\ P(X_1 = 1, X_2 = 0 ) = 1/6 \\ P(X_1 = 0, X_2 = 1 ) = 1/6 \\ P(X_1 = 1, X_2 = 1 ) = 1/3 \$\$ Tha above joint distribution in \$X_1\$, \$X_2\$ has the highest entropy for \$H(X_1 +X_2)\$. Also, by symmetry, \$X_1\$, \$X_2\$ are indentically distributed in the above distribution.</p>