38
$\begingroup$

My question may be a silly one. So I shall apologize in advance.

I was trying to use the GLOVE model pre-trained by Stanford NLP group (link). However, I noticed that my similarity results showed some negative numbers.

That immediately prompted me to look at the word-vector data file. Apparently, the values in the word vectors were allowed to be negative. That explained why I saw negative cosine similarities.

I am used to the concept of cosine similarity of frequency vectors, whose values are bounded in [0, 1]. I know for a fact that dot product and cosine function can be positive or negative, depending on the angle between vector. But I really have a hard time understanding and interpreting this negative cosine similarity.

For example, if I have a pair of words giving similarity of -0.1, are they less similar than another pair whose similarity is 0.05? How about comparing similarity of -0.9 to 0.8?

Or should I just look at the absolute value of minimal angle difference from $n\pi$? Absolute value of the scores?

Many many thanks.

$\endgroup$
4
  • 1
    $\begingroup$ Cosine similarity tag says: An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors. The only difference between the two is that in correlation deviations (moments) - which are being cross-multiplied - are from the mean, while in cosine deviations are from the original 0 - i.e. they are the values as they are. $\endgroup$
    – ttnphns
    Feb 26, 2016 at 22:24
  • 3
    $\begingroup$ (cont.) Understanding of positive or negative coefficient is the same in both instances. Negative coef. means that positive deviations/values of one vector tend to pair with negative deviations/values of the other. Whether this means the vectors are "similar" or on the contrary "highly dissimilar" is dependent on what is the meaning of positive and negative deviations/values in the data, for you. $\endgroup$
    – ttnphns
    Feb 26, 2016 at 22:28
  • $\begingroup$ @ttnphns Thank you very much for your comment! It does inspire me to think about cosine similarity in a new way. In my use case, perhaps I can think of it as a difference in end results: if the correlation of Doc A and B is negative, and an uni-topical journal X includes Doc A, then it is less likely that X includes B as well, from some mean probability. Does this interpretation sounds valid to you? $\endgroup$ Feb 26, 2016 at 22:40
  • $\begingroup$ I dare not to say because I don't know your data, meaning of values in it, and your study. $\endgroup$
    – ttnphns
    Feb 26, 2016 at 22:55

5 Answers 5

19
$\begingroup$

Let two vectors $a$ and $b$, the angle $θ$ is obtained by the scalar product and the norm of the vectors :

$$ cos(\theta) = \frac{a \cdot b}{||a|| \cdot ||b||} $$

Since the $cos(\theta)$ value is in the range $[-1,1]$ :

  • $-1$ value will indicate strongly opposite vectors
  • $0$ independent (orthogonal) vectors
  • $1$ similar (positive co-linear) vectors. Intermediate values are used to assess the degree of similarity.

Example : Let two user $U_1$ and $U_2$, and $sim(U_1, U_2)$ the similarity between these two users according to their taste for movies:

  • $sim(U_1, U_2) = 1$ if the two users have exactly the same taste (or if $U_1 = U_2$)
  • $sim(U_1, U_2) = 0$ if we do not find any correlation between the two users, e.g. if they have not seen any common movies
  • $sim(U_1, U_2) = -1$ if users have opposed tastes, e.g. if they rated the same movies in an opposite way
$\endgroup$
1
  • $\begingroup$ From a machine learning standpoint, is a negative cosine similarity benign or bad? $\endgroup$
    – moonman239
    Feb 25, 2023 at 20:49
2
$\begingroup$

Its right that cosine-similarity between frequency vectors cannot be negative as word-counts cannot be negative, but with word-embeddings (such as glove) you can have negative values.

A simplified view of Word-embedding construction is as follows: You assign each word to a random vector in R^d. Next run an optimizer that tries to nudge two similar-vectors v1 and v2 close to each other or drive two dissimilar vectors v3 and v4 further apart (as per some distance, say cosine). You run this optimization for enough iterations and at the end, you have word-embeddings with the sole criterion that similar words have closer vectors and dissimilar vectors are farther apart. The end result might leave you with some dimension-values being negative and some pairs having negative cosine similarity -- simply because the optimization process did not care about this criterion. It may have nudged some vectors well into the negative-values. The dimensions of the vectors dont correspond to word-counts, they are just some arbitrary latent concepts that admit values in -inf to +inf.

$\endgroup$
1
$\begingroup$

Do not use the absolute values, as the negative sign is not arbitrary. To acquire a cosine value between 0 and 1, you should use the following cosine function:

(R code)

cos.sim <- function(a,b) 
{
  dot_product = sum(a*b)
  anorm = sqrt(sum((a)^2))
  bnorm = sqrt(sum((b)^2))
  minx =-1
  maxx = 1
  return(((dot_product/anorm*bnorm)-minx)/(maxx-minx))
} 

(Python Code)

def cos_sim(a, b):
    """Takes 2 vectors a, b and returns the cosine similarity according 
to the definition of the dot product"""
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

minx = -1 
maxx = 1

cos_sim(row1, row2)- minx)/(maxx-minx)
```
$\endgroup$
1
  • $\begingroup$ Where do you set minx and maxx? You might apply this min-max normalization to vector dimensions, instead of calculated distance. $\endgroup$
    – emre can
    Nov 26, 2019 at 2:34
1
$\begingroup$

Cosine similarity is just like Pearson correlation, but without substracting the means. So you can compare the relative strengh of 2 cosine similarities by looking at the absolute values, just like how you would compare the absolute values of 2 Pearson correlations.

$\endgroup$
0
$\begingroup$

This article illustrates why a negative cosine similarity indicates vectors that are not similar (opposite direction in the vector space): Understanding Vector Similarity

$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Oct 8, 2023 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.