2
$\begingroup$

I have two identical probability distributions $p_1(x) = p_2(x)\ \forall x$ and wish to construct a joint distribution between them, i.e. $p(x_1,x_2) = p_1(x_1)\cdot p_2(x_2)$. Supposing I know the entropy of the univariate distribution $H(p_1) = H(p_2)$, is there a simple formula to obtain the entropy of the joint distribution H(p)?

Some sketchy calculations I did lead to to believe that $H(p) = 2 H(p_1) = 2 H(p_2)$. Is this accurate?

$\endgroup$
3
  • $\begingroup$ To be explicit, your aim was to have the two be independent? I realize it's in the title but its absence from the body text concerned me. $\endgroup$
    – Glen_b
    Nov 22, 2019 at 0:14
  • 3
    $\begingroup$ Entropy adds: this is its defining property. $\endgroup$
    – whuber
    Nov 22, 2019 at 0:17
  • 1
    $\begingroup$ Yes, independent $\endgroup$
    – jon_simon
    Nov 22, 2019 at 0:34

1 Answer 1

5
$\begingroup$

Your calculations are correct that $H(X_1, X_2) = 2H(X_1)$. More generally, if $X_1$ and $X_2$ are independent then $$H(X_1, X_2) = H(X_1) + H(X_2).$$


\begin{align*} H(X, Y) &= - E\left(\log_2\left(f_{XY}(x, y)\right)\right) \\[1.3ex] &= -E\left(\log_2\left(f_{X}(x)f_Y(y)\right)\right) &&\text{by independence} \\[1.3ex] &= -E\left(\log_2(f_X(x)) + \log_2(f_Y(y))\right) \\[1.3ex] &= -E\left(\log_2(f_X(x))\right) - E\left(\log_2(f_Y(y))\right) && \text{linearity of expectation} \\[1.3ex] &= H(X) + H(Y) && \square \end{align*}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.