38
$\begingroup$

I was using the GLOVE model which is pre-trained by Stanford NLP group (link) and noticed that my similarity results showed some negative numbers. Upon inspecting the word-vector data file, I realized that the values in the word vectors can be negative.

I am used to the concept of cosine similarity of frequency vectors, whose values are bounded in $[0, 1]$. I know that dot product and cosine function can be positive or negative, depending on the angle between vector. But what does negative cosine similarity mean in this model?

For example, if I have a pair of words giving similarity of -0.1, are they less similar than another pair whose similarity is 0.05? How about comparing similarities of -0.9 to 0.8?

Or should I just look at the absolute value of minimal angle difference from $n\pi$? Or absolute value of the scores?

$\endgroup$
4
  • 1
    $\begingroup$ Cosine similarity tag says: An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors. The only difference between the two is that in correlation deviations (moments) - which are being cross-multiplied - are from the mean, while in cosine deviations are from the original 0 - i.e. they are the values as they are. $\endgroup$
    – ttnphns
    Commented Feb 26, 2016 at 22:24
  • 3
    $\begingroup$ (cont.) Understanding of positive or negative coefficient is the same in both instances. Negative coef. means that positive deviations/values of one vector tend to pair with negative deviations/values of the other. Whether this means the vectors are "similar" or on the contrary "highly dissimilar" is dependent on what is the meaning of positive and negative deviations/values in the data, for you. $\endgroup$
    – ttnphns
    Commented Feb 26, 2016 at 22:28
  • $\begingroup$ @ttnphns Thank you very much for your comment! It does inspire me to think about cosine similarity in a new way. In my use case, perhaps I can think of it as a difference in end results: if the correlation of Doc A and B is negative, and an uni-topical journal X includes Doc A, then it is less likely that X includes B as well, from some mean probability. Does this interpretation sounds valid to you? $\endgroup$ Commented Feb 26, 2016 at 22:40
  • $\begingroup$ I dare not to say because I don't know your data, meaning of values in it, and your study. $\endgroup$
    – ttnphns
    Commented Feb 26, 2016 at 22:55

5 Answers 5

19
$\begingroup$

Given two vectors $a$ and $b$, the angle $θ$ is obtained by the scalar product and the norm of the vectors :

$$ cos(\theta) = \frac{a \cdot b}{||a|| \cdot ||b||} $$

Since $-1 \leq cos(\theta) \leq 1$,

  • $-1$ indicates strongly opposite vectors
  • $0$ independent (orthogonal) vectors
  • $1$ similar (positive co-linear) vectors. Intermediate values are used to assess the degree of similarity.

Example : Let two users $U_1$ and $U_2$, and $sim(U_1, U_2)$ be the similarity between these two users according to their taste for movies:

  • $sim(U_1, U_2) = 1$ if the two users have exactly the same taste (or if $U_1 = U_2$)
  • $sim(U_1, U_2) = 0$ if we do not find any correlation between the two users, e.g. if they have not seen any common movies
  • $sim(U_1, U_2) = -1$ if users have opposed tastes, e.g. if they rated the same movies in an opposite way
$\endgroup$
1
  • $\begingroup$ From a machine learning standpoint, is a negative cosine similarity benign or bad? $\endgroup$
    – moonman239
    Commented Feb 25, 2023 at 20:49
2
$\begingroup$

Its right that cosine-similarity between frequency vectors cannot be negative as word-counts cannot be negative, but with word-embeddings (such as glove) you can have negative values.

A simplified view of Word-embedding construction is as follows:

  • You assign each word to a random vector in $R^d$.
  • Next run an optimizer that tries to nudge two similar-vectors $v_1$ and $v_2$ close to each other or drive two dissimilar vectors $v_3$ and $v_4$ further apart (as per some distance, say cosine).
  • You run this optimization for enough iterations and at the end, you have word-embeddings with the sole criterion that similar words have closer vectors and dissimilar vectors are farther apart.

The end result might leave you with some dimension-values being negative and some pairs having negative cosine similarity — simply because the optimization process did not care about this criterion. It may nudge some vectors well into the negative-values. The dimensions of the vectors don't correspond to word-counts; they are just some arbitrary latent concepts that admit values in $-\infty$ to $+\infty$.

$\endgroup$
1
$\begingroup$

Cosine similarity is just like Pearson correlation, but without subtracting the means. So you can compare the relative strengths of 2 cosine similarities by comparing the absolute values, just like you would compare the absolute values of 2 Pearson correlations.

$\endgroup$
1
$\begingroup$

Do not use the absolute values, as the negative sign is not arbitrary. To acquire a cosine value between 0 and 1, you should use the following cosine function:

(R code)

cos.sim <- function(a,b) 
{
  dot_product = sum(a*b)
  anorm = sqrt(sum((a)^2))
  bnorm = sqrt(sum((b)^2))
  minx =-1
  maxx = 1
  return(((dot_product/anorm*bnorm)-minx)/(maxx-minx))
} 

(Python Code)

def cos_sim(a, b):
    """Takes 2 vectors a, b and returns the cosine similarity according 
to the definition of the dot product"""
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

minx = -1 
maxx = 1

cos_sim(row1, row2)- minx)/(maxx-minx)
$\endgroup$
1
  • $\begingroup$ Where do you set minx and maxx? You might apply this min-max normalization to vector dimensions, instead of calculated distance. $\endgroup$
    – emre can
    Commented Nov 26, 2019 at 2:34
0
$\begingroup$

This article illustrates why a negative cosine similarity indicates vectors that are not similar (opposite direction in the vector space): Understanding Vector Similarity

$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Oct 8, 2023 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.