1
$\begingroup$

I'm constantly confused by the ambiguous notation when discussing Entropy (H) and Mutual Information (I). For example, here's a formula from "Elements of Information Theory":

$$ I(X;Y|Z) = H(X|Z) − H(X|Y,Z) $$

In this, $H(X|Y,Z)$ can mean two things:

a) Entropy of $X$ given $(Y,Z)$ or

b) Entropy of $(X|Y)$ and $Z$

These two are completely different things!

Which of these two is correct? Logically I can see that it must be a) and not b), but this is only because of intuition. In many other formulas elsewhere, the formula is ambiguous.

Is there some rule to tell exactly what it means?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

$\text{H}(X|Y,Z)$ means the entropy of $X$ when both $Y$ and $Z$ are given. This is true for probabilities/densities, i.e. $\text{P}(X|Y,Z)$ means the distribution of $X$ given $Y,Z$. So, you're correct; it is (a) and I've never seen the usage of (b) anywhere.

Improve this answer
$\endgroup$
9
  • $\begingroup$ But how do we know for other cases? For example, what about this expression: H(X,Y|Z). Does it mean Entropy of X and (Y|Z), or is it Entropy of (X,Y), given Z? What are the rules? How do I know which is which if I see it in the future in another context where my intuition isn't good enough? $\endgroup$
    – Bhagwad Jal Park
    Commented Nov 2, 2019 at 14:23
  • $\begingroup$ $H(X,Y|Z)$ is the entropy of $(X,Y)$ given $Z$. The rules are simple. You'll assume all the variables to the right of the given sign, i.e. $\vert$, is given, and to the left of it is not given. $\endgroup$
    – gunes
    Commented Nov 2, 2019 at 14:26
  • $\begingroup$ Thanks for the explanation! In which case, how do we write: Entropy of X and (Y|Z)? $\endgroup$
    – Bhagwad Jal Park
    Commented Nov 3, 2019 at 0:18
  • $\begingroup$ I haven't seen a single reference to your situation in either your book or another one, including prob & stats. And, imho (since I can't find a discussion on the topic) it's not a possible one. When a RV is given, it's given for all the RVs. Otherwise, you're referring to different sample spaces. For example, $P(X|Y, Z)$ would mean probability of $X$ happening when $Y$ happened, and $Z$ happened (when $Y$ is not known). You won't be able to calculate it. An event either occurs or not. $\endgroup$
    – gunes
    Commented Nov 3, 2019 at 6:34
  • 1
    $\begingroup$ @BhagwadJalPark Well, I meant what you are trying to find doesn't exist in the literature because it doesn't make sense. If something (an event, a RV) is given, it is given for all the RVs in the space. For example, $P(A|B\cap C)$ means prob. of A happening if $B$ and $C$ are given. The following doesn't have a meaning: $P((A|B)\cap C)$, e.g. prob. of $A$ when $B$ happened and $C$. Did $B$ happen or not? $\endgroup$
    – gunes
    Commented Nov 3, 2019 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.