# Entropy of Joint Distribution of two IID Random Variables

I have two identical probability distributions $$p_1(x) = p_2(x)\ \forall x$$ and wish to construct a joint distribution between them, i.e. $$p(x_1,x_2) = p_1(x_1)\cdot p_2(x_2)$$. Supposing I know the entropy of the univariate distribution $$H(p_1) = H(p_2)$$, is there a simple formula to obtain the entropy of the joint distribution H(p)?

Some sketchy calculations I did lead to to believe that $$H(p) = 2 H(p_1) = 2 H(p_2)$$. Is this accurate?

• To be explicit, your aim was to have the two be independent? I realize it's in the title but its absence from the body text concerned me. Nov 22, 2019 at 0:14
• Entropy adds: this is its defining property.
– whuber
Nov 22, 2019 at 0:17
• Yes, independent Nov 22, 2019 at 0:34

## 1 Answer

Your calculations are correct that $$H(X_1, X_2) = 2H(X_1)$$. More generally, if $$X_1$$ and $$X_2$$ are independent then $$H(X_1, X_2) = H(X_1) + H(X_2).$$

\begin{align*} H(X, Y) &= - E\left(\log_2\left(f_{XY}(x, y)\right)\right) \\[1.3ex] &= -E\left(\log_2\left(f_{X}(x)f_Y(y)\right)\right) &&\text{by independence} \\[1.3ex] &= -E\left(\log_2(f_X(x)) + \log_2(f_Y(y))\right) \\[1.3ex] &= -E\left(\log_2(f_X(x))\right) - E\left(\log_2(f_Y(y))\right) && \text{linearity of expectation} \\[1.3ex] &= H(X) + H(Y) && \square \end{align*}