One way to generate word embeddings is as follows (mirror):

  1. Get a corpora, e.g. "I enjoy flying. I like NLP. I like deep learning."
  2. Build the word cooccurrence matrix from it:

enter image description here

  1. Perform SVD on $X$, and keep the first $k$ columns of U.

enter image description here

Each row of the submatrix $U_{1:|V|,1:k}$ will be the word embedding of the word that the row represents (row 1 = "I", row 2 = "like", …).

Between steps 2 and 3, pointwise mutual information is sometimes applied (e.g. A. Herbelot and E.M. Vecchi. 2015. Building a shared world: Mapping distributional to model-theoretic semantic spaces. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal.).

What are the pros and cons of applying pointwise mutual information on a word cooccurrence matrix before SVD?


1 Answer 1


according to Dan Jurafsky and James H. Martin book:

"It turns out, however, that simple frequency isn’t the best measure of association between words. One problem is that raw frequency is very skewed and not very discriminative. If we want to know what kinds of contexts are shared by apricot and pineapple but not by digital and information, we’re not going to get good discrimination from words like the, it, or they, which occur frequently with all sorts of words and aren’t informative about any particular word."

sometimes we replace this raw frequency with positive pointwise mutual information:

$$\text{PPMI}(w,c) = \max{\left(\log_{2}{\frac{P(w,c)}{P(w)P(c)}},0\right)}$$

PMI on its own shows how much it's possible to observe a word w with a context word C compare to observing them independently. In PPMI we only keep positive values of PMI. Let's think about when PMI is + or - and why we only keep negative ones:

What does positive PMI mean?

  • $\frac{P(w,c)}{(P(w)P(c))} > 1$

  • $P(w,c) > (P(w)P(c))$

  • it happens when $w$ and $c$ occur mutually more than individually like kick and ball. We'd like to keep these!

What does negative PMI mean?

  • $\frac{P(w,c)}{(P(w)P(c))} < 1$

  • $P(w,c) < (P(w)P(c))$

  • it means both of $w$ and $c$ or one of them tend to occur individually! It might indicate unreliable stats due to limited data otherwise it shows uninformative co-occurrences e.g., 'the' and 'ball'. ('the' occurs with most of the words too.)

PMI or particularly PPMI helps us to catch such situations with informative co-occurrence.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.