Intuition for GloVe word embeddings

Question

I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used.

The diagram below is used to illustrate the co-occurence ratios of two words under a certain context word $k$

The paper stated that through the ratios they were able to distinguish relevant words like "solid" and "gas" from irrelevant words such as "water" and "fashion". I don't see why that is the case.

Suppose we have $k$ = fridge, then most probably the ratio $\frac{P(fridge|ice)}{p(frdige|steam)}$ will be bigger than 1. Does this allow us to group fridge together with solid and gas ?

Paper link: https://nlp.stanford.edu/pubs/glove.pdf

Answer 1

The caption for that table:

Table 1: Co-occurrence probabilities for target words ice and steam with selected context words from a 6 billion token corpus. Only in the ratio does noise from non-discriminative words like water and fashion cancel out, so that large values (much greater than 1) correlate well with properties specific to ice, and small values (much less than 1) correlate well with properties specific of steam.

The ratio is finding words which correlate more with ice or with steam. So yes, fridge most likely will have a ratio larger than one, and the context word fridge can be used as support for ice over steam.

It may be helpful to jump ahead a bit in the development of the model, where we see that they end up choosing simplifications and desired properties that yield an objective:

$$ (\vec{\text{ice}} - \vec{\text{steam}}) \cdot \vec{k} \approx \log\left(\frac{P(k\mid\text{ice})}{P(k\mid\text{steam})}\right).$$

That is, e.g., the learned vector for fridge should generally point in the same direction as the vector from steam to ice. For fashion, however, the log is zero: the learned vector for fashion should be nearly perpendicular to the vector from steam to ice.