1
$\begingroup$

let's say that I have two continuous events (random variables) $x$ and $y$.

Then what is the integration of the pointwise mutual information between these two events with respect to $x$?

$\int$$pmi(x;y)$$dx$$=$$\int$$\log\frac{p(x, y)}{p(x)p(y)}dx$

What kind of physical meaning does this have?

$\endgroup$
1
$\begingroup$

Usually, we would consider the expectation of the pointwise mutual information. The expectation of a quantity, $\mathbb E [f(x,y)] = \int p(x,y)~f(x,y)~ dx dy $. If we put pointwise mutual information in there, we get $$\mathbb E [pmi(x;y)] = \int p(x,y) \log\frac{p(x, y)}{p(x)p(y)} dx dy = I(X;Y)$$ The expectation of the pointwise mutual information is just the mutual information, and this is really how we define pmi to begin with.

Your question asks for something a little different. To generalize a little, we could ask what the expectation of pmi is under some other distribution, like $q(x,y)$. In particular, we could imagine a $q$ that is uniform over all $x,y$ in some domain and that would recover your question (up to some normalization).

In that case we can add zero and re-arrange. $$\int q(x,y) \log\frac{p(x, y)}{p(x)p(y)} dx dy = \int q(x,y) (\log\frac{p(x, y)}{p(x)p(y)} +\log\frac{q(x, y)}{q(x)q(y)} +\log\frac{q(x) q(y)}{q(x, y)}) dx dy$$ $$ = I_q(X;Y) + D(q(x)q(y)||p(x)p(y)) - D(q(x,y)||p(x,y)) $$ Here $D$ is the KL divergence and $I_q$ denotes mutual information under the distribution $q$. If q already factorizes, i.e. $q(x,y) = q(x) q(y)$ (as might be the case if $q$ is uniform), then $I_q = 0$ and we just have $ D(q(x)q(y)||p(x)p(y)) - D(q(x) q(y)||p(x,y))$. This difference in KL divergences might be a little easier to interpret. It's like checking whether the distribution $p$ is independent with respect to a reference distribution $q$. If $p$ also factorizes, this expression will be zero. Unlike mutual information, though, certain types of dependence could exist in $p$, but this dependence relative to q might still be zero. I can't think of a natural situation where this quantity might arise, but it was interesting to consider.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.