9
$\begingroup$

I'm trying to understand the normalized form of pointwise mutual information.

$npmi = \frac{pmi(x,y)}{log(p(x,y))}$

Why does the log joint probability normalize the pointwise mutual information to be between [-1, 1]?

The point-wise mutual information is:

$pmi = log(\frac{p(x,y)}{p(x)p(y)})$

p(x,y) is bounded by [0, 1] so log(p(x,y)) is bounded by (,0]. It seems like the log(p(x,y)) should somehow balance changes in the numerator, but I don't understand exactly how. It also reminds me of entropy $h=-log(p(x))$, but again I don't understand the exact relationship.

$\endgroup$
12
$\begingroup$

From Wikipedia entry on pointwise mutual information:

Pointwise mutual information can be normalized between [-1,+1] resulting in -1 (in the limit) for never occurring together, 0 for independence, and +1 for complete co-occurrence.

Why does it happen? Well, the definition for pointwise mutual information is

$$ pmi \equiv \log \left[ \frac{p(x,y)}{p(x)p(y)} \right] = \log p(x,y) - \log p(x) - \log p(y), $$

whereas for normalized pointwise mutual information is:

$$ npmi \equiv \frac{pmi}{-\log p(x,y)} = \frac{\log[ p(x) p(y)]}{\log p(x,y)} - 1. $$

The when there are:

  • no co-occurrences, $\log p(x,y)\to -\infty$, so nmpi is -1,
  • co-occurrences at random, $\log p(x,y)= \log[p(x) p(y)]$, so nmpi is 0,
  • complete co-occurrences, $\log p(x,y)= \log p(x) = \log p(y)$, so nmpi is 1.
$\endgroup$
  • $\begingroup$ It would be a more complete answer to show why npmi is on the interval $[-1,1]$. See my proof in the other answer. $\endgroup$ – Hans Jan 15 '19 at 20:18
1
$\begingroup$

While Piotr Migdal's answer is informative in giving the examples where nmpi achieves three extreme values, it does not prove it is on the interval $[-1,1]$. Here is the inequality and its derivation. \begin{align} &\log\,p(x,y) \\ \le&\log\,p(x,y))-\log\,p(x)-\log\,p(y) \\ =&\log \frac{p(x,y)}{p(x)p(y)}=:\text{pmi}(x;y) \\ =&\log\, p(y|x)+\log\, p(y|x)-\log\,p(x,y) \\ \le&-\log\,p(x,y) \end{align} as $-\log\,p(A)\ge0$ for any event $A$. Dividing both side by the non-negative $h(x,y):=-\log\,p(x,y)$, we have $$ -1\le\text{nmpi}(x;y):=\frac{\text{mpi(x;y)}}{h(x,y)}\le1.$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.