I have some truncated normal distributions and some other distributions (like uniform distribution) that have partially common domains. I'd like to know how can I calculate the entropy for the sum of these distributions. Let X~Uniform(a,b) and Y~truncatedNormal(c,d) and Z~uniform(e,f) and a < c < e < b < d < f. Now I want to calculate the entropy of the distribution of the sum.

  • 1
    $\begingroup$ When you say "sum of these distributions" do you instead mean "the distribution of the sum of the random variables"? Are the random variables independent? Why would common domains matter (whether partial or not)? $\endgroup$ – Glen_b Jan 30 '15 at 7:51
  • $\begingroup$ @Glen_b Consider it like this: Let X has Uniform(a,b) and Y has truncated Normal (c,d) and Z has uniform(e,f) and a<c<e<b<d<f. Now I want to calculate the entropy of the distribution of the sum. $\endgroup$ – Ferra Xu Jan 30 '15 at 15:51
  • $\begingroup$ What about the remaining questions in my comment? $\endgroup$ – Glen_b Feb 1 '15 at 3:15

Independence means exactly that the joint pdf is the product of the individual pdfs.

If we are talking about probability theory, and the densities are not independent, then you have to specify how they depend on each other.

About entropy, an easy way of seeing it is through conditional entropy: the conditional entropy $H(X|Y)$ is the extra amount of entropy that you get from $X$ by knowing $Y$. Then you can write $$ H(X,Y) = H(X|Y)+H(Y) $$

of course, if $X$ depends completely from $Y$, then $H(X|Y)=0$. If instead the information of $X$ that you get from $Y$ is null, therefore $X$ and $Y$ are independent, then $H(X|Y)=X$. In this way, you can quantify the joint entropy from the individual distributions and their dependence.

The conditional entropy is computed as $$ H(X|Y)=\int dy\;p(y)\;H(X|Y=y) = -\int dy\;p(y) \int dx\;p(x|y)\;log\left(p(x|y)\right) $$

so the most convenient way of writing the dependence of the two variables would be through their conditional density, $p(x|y)$.

| cite | improve this answer | |
  • $\begingroup$ @Ferra Xu all you need is this answer and the fact that the variables are independent. $\endgroup$ – Sheridan Grant Mar 11 at 5:34
  • 2
    $\begingroup$ you can compute the entropy even if they are not independent $\endgroup$ – chuse Mar 12 at 10:53
  • $\begingroup$ Sure, OP doesn't state whether they're dependent or not (presumably independent from the way the distributions are written), and they seem to think overlapping domains introduces some difficulty when 1) that's different from independence and 2) neither overlapping domains nor independence creates a problem for computing the entropy. $\endgroup$ – Sheridan Grant Mar 12 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.