I am having trouble building some intuition about joint entropy. $H(X,Y)$ = uncertainty in the joint distribution $p(x,y)$; $H(X)$ = uncertainty in $p_x(x)$; $H(Y)$ = uncertainty in $p_y(y)$.

If H(X) is high then the distribution is more uncertain and if you know the outcome of such a distribution then you have more information! So H(X) also quantifies information.

Now we can show $H(X,Y) \leq H(X) + H(Y)$

But if you know $p(x,y)$ you can get $p_x(x)$ and $p_y(y)$ so in some sense $p(x,y)$ has more information than both $p_x(x)$ and $p_y(y)$, so shouldn't the uncertainty related to p(x,y) be more that the sum of the individual uncertainties?


as a general rule, additional information never increases the entropy, which is formally stated as:

\begin{equation} H(X|Y) \leq H(X) \, \, \, * \end{equation}

the equality holds if $X$ and $Y$ are independent, which implies $H(X|Y) = H(X)$.

This result can be used to prove the joint entropy $H(X_1, X_2, ..., X_n) \leq \sum_{i=1}^{n} H(X_i)$. To demonstrate it, consider a simple case $H(X,Y)$. According to the chain rule, we can write the join entropy as below

\begin{equation} H(X,Y) = H(X|Y) + H(Y) \end{equation}

Considering inequality $*$, $H(X|Y)$ never increases the entropy of variable $X$, and hence $H(X,Y) \leq H(X) + H(Y)$. Using induction one can generalize this result to the cases that involve more than two variables.

Hope it has helped to reduce the ambiguity (or your entropy) about the joint entropy!

| cite | improve this answer | |

There is another point of view of the Shannon entropy. Imagine you want to guess through questions what the concrete value of a variable is. For simplicity, imagine that the value can only take eight different values $\left(0,1,..., 8\right)$, and all are equally probable.

The most efficient way is to perform a binary search. First you ask whether is greater or less than 4. Then compare it against 2 or 6, and so on. In total you won't need more than three questions (which is the number of bits of this concrete distribution).

We can carry on the analogy for the case of two variables. If they are not independent, then knowing the value of one of them helps you make better guesses (in average) for the next question (this is reflected in the results pointed out by omidi). Hence, the entropy is lower, unless they are completely independent, where you need to guess their values independently. Saying that the entropy is lower means (for this concrete example) that you need to make less questions in average (i.e. more often than not you will make good guesses).

| cite | improve this answer | |

It appears you are making the thought "if more information when known, then more entropy when unknown". This is not a correct intuition, because, if the distribution is unknown, we don't even know its entropy. If the distribution is known, then entropy quantifies the information amount needed to describe uncertainty about the realization of the random variable, which remains unknown (we only know the structure surrounding this uncertainty, by knowing the distribution). Entropy does not quantify the information "present" in the distribution. On the contrary: the more information "included" in the distribution, the less information "needed" to describe uncertainty, and so the less the entropy is. Consider the uniform distribution: it contains very little information, because all possible values of the variable are equiprobable: hence it has maximum entropy among all distributions with bounded support.

As for Joint Entropy, you may think of it as follows: the joint distribution contains information about whether two variables are dependent or not, plus information sufficient to derive the marginal distributions. The marginal distributions do not contain information about whether two random variables are dependent or independent. So the joint distribution has more information, and affords us less uncertainty surrounding the random variables involved:

More information included in the distribution $\rightarrow$ less uncertainty surrounding the variables $\rightarrow$ less information needed to describe this uncertainty $\rightarrow$ less entropy.

| cite | improve this answer | |
  • $\begingroup$ Thanx, that makes things very clear. I was thinking along the lines that correlations in a distributions should decrease the uncertainty of a pair of values $(X,Y)$ and hence $H(X,Y)$ must be smaller that $H(X) + H(Y)$ . $\endgroup$ – user21455 Feb 10 '14 at 0:50
  • $\begingroup$ Yes, that's the essence. $\endgroup$ – Alecos Papadopoulos Feb 10 '14 at 1:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.