Suppose I have an operator (function) $f(\cdot)$ which takes three arguments $x,y,z$ all of which are independent random variables, and all of which I have access to the probability mass function (they are discrete random variables). Further, let's say I have an equality of the following form:

$$ w = f(x,y,z) $$ where w is also a discrete random variable (to which I also know the PMF). Let us imagine that $f(\cdot)$ is something simple like

$$ f(x,y,z) = x+y -z $$

Now, imagine we are interested in the Shannon entropy of this relation, and we apply it thus:

$$ H(w) = H(f(x,y,z)). $$

Is the following true:

\begin{align} H(w) &= H(f(x,y,z)) \\ &= H(x,y,z) \\ &= H(x)+ H(y) + H(z). \\ \end{align} Consequently,

\begin{align} H(w) &= H(x)+ H(y) + H(z). \\ \end{align}


  1. We can drop $f(\cdot)$ because the functional form does not affect the distribution nor the independence of the random variables.
  2. We can express the joint entropy as a sum because the random variables are all independent, hence any mutual information factors are zero.

Is this correct? And could anyone recommend good resources for taking the entropy of functions with random variables?

Update 1

Consider this example, we are in possession of three independent discrete random variables with this relationship: $$ Z = Y + X $$

Lets say we can approximate the entropy of $Z$ as so $H[Z]$ - the left-hand side. The right-hand side, however, takes the form:

$$ H[Y+X] $$

Now, let us suppose that both $Y$ and $X$ are normally distributed, that means we are taking the entropy of a sum. Hence, if:

$$ \begin{align} X &\sim N(\mu _{X},\sigma _{X}^{2}) \\ Y &\sim N(\mu _{Y},\sigma _{Y}^{2}) \end{align} $$ then that means that:

$$ X+Y \sim N(\mu _{X}+\mu _{Y},\sigma _{X}^{2}+\sigma _{Y}^{2}). $$

Consequently, because we know that the entropy of a Gaussian random variable is:

$$ \frac{1}{2}\log(2\pi e\sigma ^{2}) $$ this would imply that the entropy of a sum of independent Gaussian random variables is: $$ \frac{1}{2}\log \left [2\pi e \left(\sigma _{X}^{2}+\sigma _{Y}^{2} \right) \right]. $$


2 Answers 2


Regarding the first point in your reasoning, since we do not know the pmf of w, or any other information about $f(x,y,z)$ (in the general case), $f$ can not be dropped.

Regarding your second point, what can be said with certainty is that $H(w) \leq H(x) + H(y) + H(z) $, in this specific case.

Please refer to this for the proof of this: https://www2.isye.gatech.edu/~yxie77/ece587/SumRV.pdf

Hope this helps

  • $\begingroup$ Hey @Vidya - thanks for your answer that is really helpful actually. A few follow-ups if I may. We do know the pmf of $w$ in this scenario - does that help with the justification of point 1? Second, suppose the interaction form of $f(\cdot)$ is exactly $x+y - z$ - under this assumption, would reasoning point 1 be correct? $\endgroup$
    – Astrid
    Apr 27, 2018 at 13:10
  • $\begingroup$ @Astrid Even if you knew the pmf of $f(x,y,z)$, how would you reason that $H(w) = H(x,y,z)$ ? The entropy of any independent variable depends on its own symbols and their probabilities. We could possibly derive a relation between $H(w)$ and $H(x,y,z)$, but that need not necessarily be equality. $\endgroup$
    – Vidya
    Apr 27, 2018 at 15:19
  • $\begingroup$ see my updated question for an example. $\endgroup$
    – Astrid
    Apr 28, 2018 at 11:32

This isn't really answer, but I'm too much of a newbie to be allowed to comment :'(

I'm a little confused by your example. If your variables are normally distributed, then they're not discrete random variables. And if $Z=X+Y$, then although $X$ and $Y$ can be independent, $Z$ is not independent from each of $X$ and $Y$. I don't think these issues affect the main content of your question, except as far as entropy for continuous random variables is trickier than it is for discrete ones.

I'm not entirely sure what the example is supposed to show though. You've got a nice formula for the differential entropy of $Z$, but it doesn't relate it in an obvious way to the differential entropies of $X$ and $Y$. Or are you providing a counterexample to your original claim? Differential entropy is a bit messed up though anyway so you're better off looking at this: https://en.wikipedia.org/wiki/Limiting_density_of_discrete_points

  • $\begingroup$ Thanks for your comment. The entropy that I have derived can be found in most standard textbooks on this topic. Let me know if you want a reference (since I need to dig it out :). $\endgroup$
    – Astrid
    Aug 4, 2018 at 12:52
  • $\begingroup$ thanks for the reply :) sure, differential entropy is standard for continuous variables, but it's flawed since it's sometimes negative and not invariant under change of variables. The limiting density of discrete points fixes these problems and is the correct continuous analogue of discrete entropy, so might be better to use that. But which one you use doesn't really affect the answers to your questions I guess anyway :) $\endgroup$
    – DM-97
    Aug 4, 2018 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.