Cover and Thomas ("Elements of information theory") define relative entropy as:

$$H(Y|X)=-E[ \log p(Y|X)]$$

where $\log p(Y|X)$ is therefore a random variable defined, at least as I understand it, by composition with the function $p(y|x)$: $\Omega \rightarrow (X,Y) \rightarrow p(Y|X)$.

My question is if there is a probabilistic way to interpret the symbol $Y|X$ in this expression in a different way. I know for example that a meaning can be attached to the symbol $E[Y|X]$ as a random variable, but I do not know just for $Y|X$...

  • 4
    $\begingroup$ Much to the disgust of many readers of this forum, there is no random variable (or any other entity) that is denoted $Y \mid X$. There are things like the conditional distribution of $Y$ given $X=x$ but that is denoted by something like $p_{Y\mid X}(y \mid X = x)$ $\endgroup$ – Dilip Sarwate Aug 23 '17 at 19:20

In probability and statistics $Y \mid X$ is a symbol used for conditional relation between two random variables, $Y$ given $X$, e.g. conditional probability $\Pr(Y\mid X)$, conditional distribution $f_Y(y \mid X=x)$, conditional expectation $E(Y \mid X)$, or conditional entropy $H(Y \mid X)$ as in your example.

For example, $\Pr(Y = y \mid X = x )$ is a probability that $Y=y$ when $X=x$, while $E(Y | X=x)$ is the expected value for $Y$ in the cases where $X=x$, etc.

Formally, the definition of conditional probability is

$$ \Pr(Y = y \mid X = x) = \frac{\Pr( Y=y\ \cap X=x)}{\Pr(X=x)} $$

So it's an operation on the "slice" of the joint distribution of $Y$ and $X$ for fixed $X=x$. For more details and multiple worked examples check the linked Wikipedia articles.

| cite | improve this answer | |
  • 3
    $\begingroup$ The classical definition of conditional probability (which is this one) comes with the asterisk that it only works when $Pr(X = x) > 0$. $\endgroup$ – Kodiologist Aug 23 '17 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.