The mutual information of two random variables $X$ and $Y$ tells us how much knowing one reduces uncertainty about the other (roughly speaking). Mathematically, the mutual information is $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)}.$$ I was wondering whether it is meaningful to consider the mutual information for specific values of $X$ and $Y$, i.e. $$I(X=x;Y=y) = p(x,y) \log \frac{p(x,y)}{p(x)p(y)}.$$ If so, what's the right interpretation of this quantity?


1 Answer 1


I found a fairly good discussion (which I'll summarize here) of my question on p. 170 of this book. Apparently the quantity I'm interested in is known as local mutual information or point-wise mutual information and is defined as

$$i(x;y) = \log \frac{p(x \mid y)}{p(x)}.$$

It can be shown that

$$i(x;y) = h(x) - h(x \mid y) = h(y) - h(y \mid x)$$ and $$I(X \mid Y) = \mathrm{E}[i(x;y)],$$ where $h$ measures the Shannon information content and $I$ is the mutual information. Intuitively $i(x;y)$ quantifies “the amount of information provided by the occurrence of the event represented by $y$ about the occurrence of the event represented by $x$" (Fano, "Transmission of information: a statistical theory of communications," 1961).

Note that $i(x;y)$ is positive iff $p(x \mid y) > p(x)$. As a concrete example, consider the two "random variables" Weather and Forecast which can each take on two possible values: rain or shine. Let's suppose that forecasters generally do a good job so that $$P(Weather = rain \mid Forecast = shine) < P(Weather = rain).$$ This implies that $$i(Weather = rain \texttt{ };\texttt{ } Forecast = shine) < 0.$$ We consider it less likely to rain when the forecast says shine than we would if we didn't hear the forecast at all. If it does in fact rain, then the forecast was misleading or misinformative; this is what is meant by $i(x;y) < 0.$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.