# How does the log(p(x,y)) normalize the point-wise mutual information?

I'm trying to understand the normalized form of pointwise mutual information.

$npmi = \frac{pmi(x,y)}{log(p(x,y))}$

Why does the log joint probability normalize the pointwise mutual information to be between [-1, 1]?

The point-wise mutual information is:

$pmi = log(\frac{p(x,y)}{p(x)p(y)})$

p(x,y) is bounded by [0, 1] so log(p(x,y)) is bounded by (,0]. It seems like the log(p(x,y)) should somehow balance changes in the numerator, but I don't understand exactly how. It also reminds me of entropy $h=-log(p(x))$, but again I don't understand the exact relationship.

From Wikipedia entry on pointwise mutual information:

Pointwise mutual information can be normalized between [-1,+1] resulting in -1 (in the limit) for never occurring together, 0 for independence, and +1 for complete co-occurrence.

Why does it happen? Well, the definition for pointwise mutual information is

$$pmi \equiv \log \left[ \frac{p(x,y)}{p(x)p(y)} \right] = \log p(x,y) - \log p(x) - \log p(y),$$

whereas for normalized pointwise mutual information is:

$$npmi \equiv \frac{pmi}{-\log p(x,y)} = \frac{\log[ p(x) p(y)]}{\log p(x,y)} - 1.$$

The when there are:

• no co-occurrences, $\log p(x,y)\to -\infty$, so nmpi is -1,
• co-occurrences at random, $\log p(x,y)= \log[p(x) p(y)]$, so nmpi is 0,
• complete co-occurrences, $\log p(x,y)= \log p(x) = \log p(y)$, so nmpi is 1.
• It would be a more complete answer to show why npmi is on the interval $[-1,1]$. See my proof in the other answer.
– Hans
Commented Jan 15, 2019 at 20:18

While Piotr Migdal's answer is informative in giving the examples where nmpi achieves three extreme values, it does not prove it is on the interval $$[-1,1]$$. Here is the inequality and its derivation. \begin{align} &\log\,p(x,y) \\ \le&\log\,p(x,y)-\log\,p(x)-\log\,p(y) \\ =&\log \frac{p(x,y)}{p(x)p(y)}=:\text{pmi}(x;y) \\ =& \log \left( \frac{p(x,y)}{p(x)} \, \frac{p(x,y)}{p(y)} \,\frac{1}{p(x,y)} \right) \\ =& \log \frac{p(x|y)\,p(y|x)}{p(x,y)} \\ \leq&-\log\,p(x,y) \end{align} $$\therefore \log\,p(x,y) \leq \text{pmi}(x;y) \le-\log\,p(x,y)$$

Dividing both side by the non-negative $$h(x,y):=-\log\,p(x,y)$$, we have $$-1\le\text{npmi}(x;y):=\frac{\text{pmi(x;y)}}{h(x,y)}\le1.$$

• The last step should use $\leq$ instead of $=$. Commented May 7 at 16:38
• @learner: Which last step?
– Hans
Commented May 8 at 23:18
• it should be $\log p(x \mid y) + \log p(y \mid x) - \log p(x, y) \leq \log p(x, y)$. Commented May 10 at 7:58
• @learner: You are right. Corrected.
– Hans
Commented May 10 at 17:41