One way to define the mutual information is

$I(X;Y) = H(X) - H(X|Y)$

I have found it useful to look the related quantity

$?(X;Y=y) = H(X) - H(X|Y=y)$

That is, we look at how much the entropy of $X$ is decreased given a particular outcome $y$ for $Y$.

It is not hard to see that the mutual information is regained on expectation on $Y$, so that we have

$E_Y[?(X;Y=y)] \\ = \sum_y p(y)(H(X) - H(X|Y=y)) \\ = H(X) - \sum_y p(y)H(X|Y=y) \\ = H(X) - H(X|Y) \\ = I(X;Y)$

My question: does my $?(X;Y=y)$ function have a name? Or a standardized notation?

Note it's not the same as pointwise mutual information: rather, I think this would be the expectation of pointwise mutual information, but only on $X$ (rather than both variables). So it's in between the regular mutual information and the pointwise version.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.