# Difference between mutual and conditional information

I have studied about the basic probability theory. Now I am studing about entropy from information theory point of view from Bose's Information Theory, Coding and Cryptography.
I have not understood the difference between mutual and conditional information.

Suppose there are two random variabes $$X\;,Y$$ with outcomes $$x_i$$ where $$i=1,2,...,n$$ and $$y_j$$ where $$j=1,2,...m$$.
The Mutual Information $$I(x_i;y_j)$$ between $$x_i$$ and $$y_j$$ is defined as $$I(x_i;y_j)=\log \frac{P(x_i|y_j)}{P(x_i)}$$.

The conditional information between $$x_i$$ and $$y_j$$ is defined as $$I(x_i|y_j)=\log \frac{1}{P(x_i|y_j)}$$

They give an example for mutual information in the book. We can see that if $$p=0$$ (meaning the channel is ideal), then knowing the input is $$0$$ or $$1$$ gives us all the information about the output (which is also $$0$$ or $$1$$ respectively).

For $$p=0$$, $$I(Y=0;X=0)=\log_2 2(1-p)=1$$.

Also, $$I(Y=0|X=0)=-\log_2(P(Y=0|X=0))=-\log_2(1-p)=-\log_2(1)=0$$

Q-1 I am not able to understand intuitively what the difference is between $$I(Y=0;X=0)=1$$ and $$I(Y=0|X=0)=0$$. What is the difference between mutual and conditional information?

Q-2 For $$p<0.5$$, $$I(Y=0;X=0)=\log_22(1-p)<0$$. What does the negative information mean?

• Note that most sources would call $\operatorname{I}(x_0; y_0)$ the pointwise mutual information. Mar 31, 2021 at 19:18

I think using $$\operatorname{I}$$ for both conditional information and pointwise mutual information might confuse things, so I'll use $$h$$ for conditional information and $$\operatorname{pmi}$$ for PMI.

## Q1: pointwise mutual information versus conditional entropy

The 'conditional information' $$h(y_0 \mid x_0)$$ is a measure of uncertainty: how surprised are we that $$Y = 0$$ given that $$X = 0$$? In a noiseless channel, we're not surprised at all, which is why the value is $$0$$. If we already know that $$X = 0$$ in a noiseless channel, then finding out that $$Y=0$$ gives no additional information. As the noise increases, so will our surprise if $$Y=0$$ given that $$X=0$$.

By contrast, $$\operatorname{pmi}(x_0; y_0)$$ is a measure of the shared information content. Given what we already know about the individual events $$x_0$$ and $$y_0$$, how much does one tell us about the other?

The two quantities can be related by this equality: $$\operatorname{pmi}(x_0; y_0) = h(y_0) - h(y_0 \mid x_0)$$. Knowing that the event $$x_0$$ occurs may make $$y_0$$ more or less surprising. The $$\operatorname{pmi}$$ function quantifies this change in surprise in bits.

## Q2: meaning of negative PMI

When the pointwise mutual information is negative, this means that $$x$$ and $$y$$ co-occur less frequently than would be expected if they were independent. This can be made clearer by rewriting the formula:

$$\operatorname{pmi}(x_0; y_0) = \log \frac{p(y_0 \mid x_0)}{p(y_0)} = \log \frac{p(y_0 \mid x_0) p(x_0)}{p(y_0)p(x_0)} = \log \frac{p(y_0, x_0)}{p(y_0)p(x_0)}$$

The fraction is less than 1 when the two events don't tend to co-occur as often as if they were independent; the log of the fraction is thus negative.

Because of the very-high noise in the channel, we don't expect $$x$$ and $$y$$ to have the same value; it's more likely expect the channel to corrupt $$x$$.

• Thanks for the reply, the conditional information $h(y_o,x_o)$ is the surprise we feel (or the information) if we come to know the information $x_o$ first. If the channel is ideal or toward ideal, then if $x=0$, there is high probability that $y=0$, so we get less suprise after knowing $x=0$. But still intuitively I am not able to understand about pointwise mutual information. It is a measure of shared information content. What is meant by that? Can you please explain it in somewhat detail.
– Iti
Apr 1, 2021 at 6:14
• Knowing $X$ reduces your uncertainty about $Y$ by $\operatorname{pmi}$ bits. The same is true about the uncertainty in $X$ from knowing $Y$. That’s what the PMI is measuring. Apr 3, 2021 at 15:22