I have studied about the basic probability theory. Now I am studing about entropy from information theory point of view from Bose's Information Theory, Coding and Cryptography.
I have not understood the difference between mutual and conditional information.

Suppose there are two random variabes $X\;,Y$ with outcomes $x_i$ where $i=1,2,...,n$ and $y_j$ where $j=1,2,...m$.
The Mutual Information $I(x_i;y_j)$ between $x_i$ and $y_j$ is defined as $I(x_i;y_j)=\log \frac{P(x_i|y_j)}{P(x_i)}$.

The conditional information between $x_i$ and $y_j$ is defined as $I(x_i|y_j)=\log \frac{1}{P(x_i|y_j)}$

They give an example for mutual information in the book. Screenshot of Example 1.3

We can see that if $p=0$ (meaning the channel is ideal), then knowing the input is $0$ or $1$ gives us all the information about the output (which is also $0$ or $1$ respectively).

For $p=0$, $I(Y=0;X=0)=\log_2 2(1-p)=1$.

Also, $I(Y=0|X=0)=-\log_2(P(Y=0|X=0))=-\log_2(1-p)=-\log_2(1)=0$

Q-1 I am not able to understand intuitively what the difference is between $I(Y=0;X=0)=1$ and $I(Y=0|X=0)=0$. What is the difference between mutual and conditional information?

Q-2 For $p<0.5$, $I(Y=0;X=0)=\log_22(1-p)<0$. What does the negative information mean?

  • $\begingroup$ Note that most sources would call $\operatorname{I}(x_0; y_0)$ the pointwise mutual information. $\endgroup$ – Arya McCarthy Mar 31 at 19:18

I think using $\operatorname{I}$ for both conditional information and pointwise mutual information might confuse things, so I'll use $h$ for conditional information and $\operatorname{pmi}$ for PMI.

Q1: pointwise mutual information versus conditional entropy

The 'conditional information' $h(y_0 \mid x_0)$ is a measure of uncertainty: how surprised are we that $Y = 0$ given that $X = 0$? In a noiseless channel, we're not surprised at all, which is why the value is $0$. If we already know that $X = 0$ in a noiseless channel, then finding out that $Y=0$ gives no additional information. As the noise increases, so will our surprise if $Y=0$ given that $X=0$.

By contrast, $\operatorname{pmi}(x_0; y_0)$ is a measure of the shared information content. Given what we already know about the individual events $x_0$ and $y_0$, how much does one tell us about the other?

The two quantities can be related by this equality: $\operatorname{pmi}(x_0; y_0) = h(y_0) - h(y_0 \mid x_0)$. Knowing that the event $x_0$ occurs may make $y_0$ more or less surprising. The $\operatorname{pmi}$ function quantifies this change in surprise in bits.

Q2: meaning of negative PMI

When the pointwise mutual information is negative, this means that $x$ and $y$ co-occur less frequently than would be expected if they were independent. This can be made clearer by rewriting the formula:

$$ \operatorname{pmi}(x_0; y_0) = \log \frac{p(y_0 \mid x_0)}{p(y_0)} = \log \frac{p(y_0 \mid x_0) p(x_0)}{p(y_0)p(x_0)} = \log \frac{p(y_0, x_0)}{p(y_0)p(x_0)} $$

The fraction is less than 1 when the two events don't tend to co-occur as often as if they were independent; the log of the fraction is thus negative.

Because of the very-high noise in the channel, we don't expect $x$ and $y$ to have the same value; it's more likely expect the channel to corrupt $x$.

  • $\begingroup$ Thanks for the reply, the conditional information $h(y_o,x_o)$ is the surprise we feel (or the information) if we come to know the information $x_o$ first. If the channel is ideal or toward ideal, then if $x=0$, there is high probability that $y=0$, so we get less suprise after knowing $x=0$. But still intuitively I am not able to understand about pointwise mutual information. It is a measure of shared information content. What is meant by that? Can you please explain it in somewhat detail. $\endgroup$ – Iti Apr 1 at 6:14
  • $\begingroup$ Knowing $X$ reduces your uncertainty about $Y$ by $\operatorname{pmi}$ bits. The same is true about the uncertainty in $X$ from knowing $Y$. That’s what the PMI is measuring. $\endgroup$ – Arya McCarthy Apr 3 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.