0
$\begingroup$

One way to define the mutual information is

$I(X;Y) = H(X) - H(X|Y)$

I have found it useful to look the related quantity

$?(X;Y=y) = H(X) - H(X|Y=y)$

That is, we look at how much the entropy of $X$ is decreased given a particular outcome $y$ for $Y$.

It is not hard to see that the mutual information is regained on expectation on $Y$, so that we have

$E_Y[?(X;Y=y)] \\ = \sum_y p(y)(H(X) - H(X|Y=y)) \\ = H(X) - \sum_y p(y)H(X|Y=y) \\ = H(X) - H(X|Y) \\ = I(X;Y)$

My question: does my $?(X;Y=y)$ function have a name? Or a standardized notation?

Note it's not the same as pointwise mutual information: rather, I think this would be the expectation of pointwise mutual information, but only on $X$ (rather than both variables). So it's in between the regular mutual information and the pointwise version.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.