I have distributions of N random variables (supposed conditionally independent) consequently, the joint distribution is the multiplication of all the distributions.

I want to compute the joint entropy (of the N variables) without using the joint distribution because of the very high dimensionality.

Is it possible to compute the joint entropy directly from the marginal distributions?


1 Answer 1


If they are (conditionally) independent, as you say, then the joint entropy (conditioned on the same thing as above) is the sum of the marginal entropies.

This follows immediately from the definition of entropy (e.g., for continuous variables, but the derivation is the same for discrete ones): $$ \mathcal{H}[p] = -\int p(\mathbf{x}) \log p(\mathbf{x}) d\mathbf{x} = -\int \prod_i p_i(x_i) \log \prod_j p_j(x_j) d \mathbf{x} $$ where I used the independence assumption, and considering that the logarithm of a product is the sum of the logarithms, $$ \mathcal{H}[p] = - \sum_j \int \prod_i p_i(x_i) \log p_j(x_j) d \mathbf{x} = \sum_j -\int p_j(x_j) \log p_j(x_j) d x_i $$ where the last passage follows from the fact that all the probability distributions different than $j$ inside the integral integrate to 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.