7
$\begingroup$

The joint entropy is the amount of information we get when we observe X and Y at the same time, but what would happen if we don't observe them at the same time.

For example, when i toss a coin, if i got tails i will only observe the variable X, but if i got heads i will only observe the variable Y. How could i find the entropy?

$\endgroup$

1 Answer 1

17
$\begingroup$

Entropy (joint entropy included), is a property of the distribution that a random variable follows. The available sample (and hence the timing of observation) plays no role in it.

Copying for Cover & Thomas, the joint entropy $H(X,Y)$ of two discrete random variables $X, Y,$ with joint distribution $p(x,y)$, is defined as

$$H(X,Y) = - \sum_{S_X}\sum_{S_Y}p(x,y)\log p(x,y) $$

Examine the expression: the sums are taken over all possible values of $X$ and $Y$, i.e. over all the values that belong to the support of each r.v. ($S_X$ and $S_Y$ respectively), irrespective of whether some of these values may not materialize or be observed in a sample. What we actually observe, or when, plays no role, in calculating entropy, and joint entropy in particular.

Turning to your specific example: The side of a coin itself can not be modeled as a random variable. A random variable maps events into real numbers. The side of a coin is not an event. Observing one of the two sides is an event. Not observing a side, is an event. So let's define a random variable $X$ by "$X$ takes the value $1$ if heads is observed, $0$ otherwise". And define $Y$ by "$Y$ takes the value $1$ if tails is observed, $0$ otherwise". Assume the coin is fair. The joint distribution of these two random variables is then described by $$\begin{align} P(X=1,Y=1) &= 0 \\ P(X=1,Y=0) &= 0.5 \\ P(X=0,Y=1) &= 0.5 \\ P(X=0,Y=0) &= 0 \end{align}$$

Note that the numerical mapping we chose (the zero/one values) does not play, as numbers go, any decisive part in the probabilities assigned -we could have chosen a 5/6 mapping for $X$ and a 56/89 mapping for $Y$ (or whatever) -the allocation of probabilities in the joint distribution would have been the same (it is the underlying structure of events that is the critical factor).
Next, as always, we consider the distribution at non-zero values, so

$$H(X,Y) = - 0.5\log(0.5) - 0.5\log(0.5) $$

and using base-2 for the logarithm we get

$$H(X,Y) = - 0.5(-1) - 0.5(-1) = 1 $$

Finally, you can easily find that the entropy of $X$ (and likewise for $Y$) is $$H(X) = - \sum_{S_X}p(x)\log p(x) = - 0.5(-1) - 0.5(-1) = 1 $$

So in this case $H(X,Y) = H(X) = H(Y)$. But the general expression for the decomposition of joint entropy is

$$H(X,Y) = H(X) + H(Y\mid X) = H(Y) + H(X\mid Y)$$

where $H(Y\mid X)$ and $H(X\mid Y)$ are conditional entropies. Then we conclude that $H(Y\mid X) = H(X\mid Y) = 0$ in this case. The intuition is straightforward: given $X$ what has happened to $Y$ is certain (and likewise in reverse), so conditional entropy is zero.

$\endgroup$
1
  • 1
    $\begingroup$ +1 for the detailed answer with examples on a basic question! $\endgroup$
    – Haitao Du
    Commented Mar 27, 2018 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.