I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used.

The diagram below is used to illustrate the co-occurence ratios of two words under a certain context word $k$

enter image description here

The paper stated that through the ratios they were able to distinguish relevant words like "solid" and "gas" from irrelevant words such as "water" and "fashion". I don't see why that is the case.

Suppose we have $k$ = fridge, then most probably the ratio $\frac{P(fridge|ice)}{p(frdige|steam)}$ will be bigger than 1. Does this allow us to group fridge together with solid and gas ?

Paper link: https://nlp.stanford.edu/pubs/glove.pdf


The caption for that table:

Table 1: Co-occurrence probabilities for target words ice and steam with selected context words from a 6 billion token corpus. Only in the ratio does noise from non-discriminative words like water and fashion cancel out, so that large values (much greater than 1) correlate well with properties specific to ice, and small values (much less than 1) correlate well with properties specific of steam.

The ratio is finding words which correlate more with ice or with steam. So yes, fridge most likely will have a ratio larger than one, and the context word fridge can be used as support for ice over steam.

It may be helpful to jump ahead a bit in the development of the model, where we see that they end up choosing simplifications and desired properties that yield an objective:

$$ (\vec{\text{ice}} - \vec{\text{steam}}) \cdot \vec{k} \approx \log\left(\frac{P(k\mid\text{ice})}{P(k\mid\text{steam})}\right).$$

That is, e.g., the learned vector for fridge should generally point in the same direction as the vector from steam to ice. For fashion, however, the log is zero: the learned vector for fashion should be nearly perpendicular to the vector from steam to ice.

  • $\begingroup$ Thanks for your reply Ben. It helped me understand a little better. I'm still not very clear why they decided to introduce a 3rd variable $k$. We can also predict the log probabilities for a word-word pair, as in what word2vec does I believe. $\endgroup$
    – calveeen
    Feb 4 '21 at 3:13
  • $\begingroup$ Also from the paper: "The relationship of these words can be examined by studying the ratio of their co-occurrence probabilities with various probe words, $k$. For words $k$ related to ice but not steam, say $k=\text{solid}$, we expect the ratio $P_{ik}/P_{jk}$ will be large." That's their motivation, but I couldn't say whether/why it's better than e.g. word2vec. After all, the two models seem comparably strong in applications. $\endgroup$ Feb 4 '21 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.