# Interpreting negative cosine similarity

I was using the GLOVE model which is pre-trained by Stanford NLP group (link) and noticed that my similarity results showed some negative numbers. Upon inspecting the word-vector data file, I realized that the values in the word vectors can be negative.

I am used to the concept of cosine similarity of frequency vectors, whose values are bounded in $$[0, 1]$$. I know that dot product and cosine function can be positive or negative, depending on the angle between vector. But what does negative cosine similarity mean in this model?

For example, if I have a pair of words giving similarity of -0.1, are they less similar than another pair whose similarity is 0.05? How about comparing similarities of -0.9 to 0.8?

Or should I just look at the absolute value of minimal angle difference from $$n\pi$$? Or absolute value of the scores?

• Cosine similarity tag says: An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors. The only difference between the two is that in correlation deviations (moments) - which are being cross-multiplied - are from the mean, while in cosine deviations are from the original 0 - i.e. they are the values as they are. Commented Feb 26, 2016 at 22:24
• (cont.) Understanding of positive or negative coefficient is the same in both instances. Negative coef. means that positive deviations/values of one vector tend to pair with negative deviations/values of the other. Whether this means the vectors are "similar" or on the contrary "highly dissimilar" is dependent on what is the meaning of positive and negative deviations/values in the data, for you. Commented Feb 26, 2016 at 22:28
• @ttnphns Thank you very much for your comment! It does inspire me to think about cosine similarity in a new way. In my use case, perhaps I can think of it as a difference in end results: if the correlation of Doc A and B is negative, and an uni-topical journal X includes Doc A, then it is less likely that X includes B as well, from some mean probability. Does this interpretation sounds valid to you? Commented Feb 26, 2016 at 22:40
• I dare not to say because I don't know your data, meaning of values in it, and your study. Commented Feb 26, 2016 at 22:55

Given two vectors $$a$$ and $$b$$, the angle $$θ$$ is obtained by the scalar product and the norm of the vectors :

$$cos(\theta) = \frac{a \cdot b}{||a|| \cdot ||b||}$$

Since $$-1 \leq cos(\theta) \leq 1$$,

• $$-1$$ indicates strongly opposite vectors
• $$0$$ independent (orthogonal) vectors
• $$1$$ similar (positive co-linear) vectors. Intermediate values are used to assess the degree of similarity.

Example : Let two users $$U_1$$ and $$U_2$$, and $$sim(U_1, U_2)$$ be the similarity between these two users according to their taste for movies:

• $$sim(U_1, U_2) = 1$$ if the two users have exactly the same taste (or if $$U_1 = U_2$$)
• $$sim(U_1, U_2) = 0$$ if we do not find any correlation between the two users, e.g. if they have not seen any common movies
• $$sim(U_1, U_2) = -1$$ if users have opposed tastes, e.g. if they rated the same movies in an opposite way
• From a machine learning standpoint, is a negative cosine similarity benign or bad? Commented Feb 25, 2023 at 20:49

Its right that cosine-similarity between frequency vectors cannot be negative as word-counts cannot be negative, but with word-embeddings (such as glove) you can have negative values.

A simplified view of Word-embedding construction is as follows:

• You assign each word to a random vector in $$R^d$$.
• Next run an optimizer that tries to nudge two similar-vectors $$v_1$$ and $$v_2$$ close to each other or drive two dissimilar vectors $$v_3$$ and $$v_4$$ further apart (as per some distance, say cosine).
• You run this optimization for enough iterations and at the end, you have word-embeddings with the sole criterion that similar words have closer vectors and dissimilar vectors are farther apart.

The end result might leave you with some dimension-values being negative and some pairs having negative cosine similarity — simply because the optimization process did not care about this criterion. It may nudge some vectors well into the negative-values. The dimensions of the vectors don't correspond to word-counts; they are just some arbitrary latent concepts that admit values in $$-\infty$$ to $$+\infty$$.

Cosine similarity is just like Pearson correlation, but without subtracting the means. So you can compare the relative strengths of 2 cosine similarities by comparing the absolute values, just like you would compare the absolute values of 2 Pearson correlations.

Do not use the absolute values, as the negative sign is not arbitrary. To acquire a cosine value between 0 and 1, you should use the following cosine function:

(R code)

cos.sim <- function(a,b)
{
dot_product = sum(a*b)
anorm = sqrt(sum((a)^2))
bnorm = sqrt(sum((b)^2))
minx =-1
maxx = 1
return(((dot_product/anorm*bnorm)-minx)/(maxx-minx))
}


(Python Code)

def cos_sim(a, b):
"""Takes 2 vectors a, b and returns the cosine similarity according
to the definition of the dot product"""
dot_product = np.dot(a, b)
norm_a = np.linalg.norm(a)
norm_b = np.linalg.norm(b)
return dot_product / (norm_a * norm_b)

minx = -1
maxx = 1

cos_sim(row1, row2)- minx)/(maxx-minx)

• Where do you set minx and maxx? You might apply this min-max normalization to vector dimensions, instead of calculated distance. Commented Nov 26, 2019 at 2:34

This article illustrates why a negative cosine similarity indicates vectors that are not similar (opposite direction in the vector space): Understanding Vector Similarity