Interpreting negative cosine similarity

Question

My question may be a silly one. So I shall apologize in advance.

I was trying to use the GLOVE model pre-trained by Stanford NLP group (link). However, I noticed that my similarity results showed some negative numbers.

That immediately prompted me to look at the word-vector data file. Apparently, the values in the word vectors were allowed to be negative. That explained why I saw negative cosine similarities.

I am used to the concept of cosine similarity of frequency vectors, whose values are bounded in [0, 1]. I know for a fact that dot product and cosine function can be positive or negative, depending on the angle between vector. But I really have a hard time understanding and interpreting this negative cosine similarity.

For example, if I have a pair of words giving similarity of -0.1, are they less similar than another pair whose similarity is 0.05? How about comparing similarity of -0.9 to 0.8?

Or should I just look at the absolute value of minimal angle difference from $n\pi$? Absolute value of the scores?

Many many thanks.

Answer 1

Let two vectors $a$ and $b$, the angle $θ$ is obtained by the scalar product and the norm of the vectors :

$$ cos(\theta) = \frac{a \cdot b}{||a|| \cdot ||b||} $$

Since the $cos(\theta)$ value is in the range $[-1,1]$ :

• $-1$ value will indicate strongly opposite vectors
• $0$ independent (orthogonal) vectors
• $1$ similar (positive co-linear) vectors. Intermediate values are used to assess the degree of similarity.

Example : Let two user $U_1$ and $U_2$, and $sim(U_1, U_2)$ the similarity between these two users according to their taste for movies:

• $sim(U_1, U_2) = 1$ if the two users have exactly the same taste (or if $U_1 = U_2$)
• $sim(U_1, U_2) = 0$ if we do not find any correlation between the two users, e.g. if they have not seen any common movies
• $sim(U_1, U_2) = -1$ if users have opposed tastes, e.g. if they rated the same movies in an opposite way

Answer 2

Its right that cosine-similarity between frequency vectors cannot be negative as word-counts cannot be negative, but with word-embeddings (such as glove) you can have negative values.

A simplified view of Word-embedding construction is as follows: You assign each word to a random vector in R^d. Next run an optimizer that tries to nudge two similar-vectors v1 and v2 close to each other or drive two dissimilar vectors v3 and v4 further apart (as per some distance, say cosine). You run this optimization for enough iterations and at the end, you have word-embeddings with the sole criterion that similar words have closer vectors and dissimilar vectors are farther apart. The end result might leave you with some dimension-values being negative and some pairs having negative cosine similarity -- simply because the optimization process did not care about this criterion. It may have nudged some vectors well into the negative-values. The dimensions of the vectors dont correspond to word-counts, they are just some arbitrary latent concepts that admit values in -inf to +inf.

Answer 3

Do not use the absolute values, as the negative sign is not arbitrary. To acquire a cosine value between 0 and 1, you should use the following cosine function:

(R code)

cos.sim <- function(a,b) 
{
  dot_product = sum(a*b)
  anorm = sqrt(sum((a)^2))
  bnorm = sqrt(sum((b)^2))
  minx =-1
  maxx = 1
  return(((dot_product/anorm*bnorm)-minx)/(maxx-minx))
} 

(Python Code)

def cos_sim(a, b):
    """Takes 2 vectors a, b and returns the cosine similarity according 
to the definition of the dot product"""
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

minx = -1 
maxx = 1

cos_sim(row1, row2)- minx)/(maxx-minx)
```

Answer 4

Cosine similarity is just like Pearson correlation, but without substracting the means. So you can compare the relative strengh of 2 cosine similarities by looking at the absolute values, just like how you would compare the absolute values of 2 Pearson correlations.

Answer 5

This article illustrates why a negative cosine similarity indicates vectors that are not similar (opposite direction in the vector space): Understanding Vector Similarity