1
$\begingroup$

Haven't used cosine similarity much in the past so getting into it now. Seeing results that are counterintuitive and would love your help making sense of them.

Assume these simple vectors:

a = [1,2,3,4]
b = [1,1,1,1]
c = [1,2,1,1]

The cosine similarity of a and b is 0.912.
The cosine similarity of a and c is 0.828.

So even though c is more similar to a than b (since c is like b only it has the same value as a on the second item), the cosine similarity is lower. Can't wrap my head around that.

Apologies for the n00b question, hope you can help!

$\endgroup$
3
  • 1
    $\begingroup$ Cosine similarity measures the angle between two vectors, not the displacement. Comparing vectors with unit length, instead of 3 different lengths, may be more helpful to forming intuition. $\endgroup$ – Sycorax Feb 9 at 1:32
  • $\begingroup$ Thanks @Sycorax. I guess my confusion is that if we think about vectors a and c, the angle they have on their 2nd dimension is 0, whereas the angle on the 2nd dimension between a and b is >0. Does that make sense? $\endgroup$ – Optimesh Feb 9 at 13:12
  • 1
    $\begingroup$ Cosine similarity doesn’t compute angles that way $\endgroup$ – Sycorax Feb 9 at 13:22
2
$\begingroup$

I got intersted in this question and here's a visual representation of why a is more similar to b than c. Thanks to @Sycorax's comment that put me in the right direction. The code is R but it should be easy to translate to python or something else.

Consider these 2D vectors (in 4D things are the same but cannot be visualised):

a <- c(1, 2)
b <- c(3, 4)
c <- c(1, 0.5)

One may expect a to be more similar to c than to b. However, the cosine similarities are:

cos_sim <- function(A, B) {
    xs <- sum(A * B) / (sqrt(sum(A^2)) * sqrt(sum(B^2)))
    return(xs)
}

cos_sim(a, b) # 0.98
cos_sim(a, c) # 0.8

This is because the angle between a and b is narrower than between a and c:

enter image description here

We can normalise the vectors to unit length and check the cosine similarities don't change:

normalise <- function(x) {
    x / sqrt(sum(x^2))
}

A <- normalise(a)
B <- normalise(b)
C <- normalise(c)

cos_sim(A, B) # Same as above
cos_sim(A, C)

enter image description here

Code for plots:

plot(x= 0:4, y= 0:4, type= 'n', bty= 'l', xlab= '', ylab= '')
grid()
arrows(x0= 0, y0= 0, x1= a[1], y1= a[2], col= 'blue')
arrows(x0= 0, y0= 0, x1= b[1], y1= b[2], col= 'red')
arrows(x0= 0, y0= 0, x1= c[1], y1= c[2], col= 'black')
text(x= c(a[1], b[1], c[1]) + 0.1, y= c(a[2], b[2], c[2]) + 0.1, col= c('blue', 'red', 'black'), labels= c('a', 'b', 'c'), cex= 2)

plot(x= 0:4, y= 0:4, type= 'n', bty= 'l', xlab= '', ylab= '')
grid()
arrows(x0= 0, y0= 0, x1= A[1], y1= A[2], col= 'blue')
arrows(x0= 0, y0= 0, x1= B[1], y1= B[2], col= 'red')
arrows(x0= 0, y0= 0, x1= C[1], y1= C[2], col= 'black')
text(x= c(A[1], B[1], C[1]) + 0.1, y= c(A[2], B[2], C[2]) + 0.1, col= c('blue', 'red', 'black'), labels= c('A', 'B', 'C'), cex= 2)
$\endgroup$
4
  • $\begingroup$ Thank you very much @dariober, this is very thoughtful of you, but unfortunately doesn't help me with intuition, because unlike your choice of b and c, my b and c are identical in all dimensions but one, and in the one dimension c has a zero angle with a, which is not the case between b and a. Does that make sense? Thanks again though, much appreciated. $\endgroup$ – Optimesh Feb 9 at 13:19
  • 1
    $\begingroup$ Opti, it's hard to make sense of your comment, because the angle between c and a obviously is not zero. This answer correctly explains and illustrates the distinction between the cosine distance and the magnitudes of differences in the components of vectors. $\endgroup$ – whuber Feb 9 at 14:52
  • $\begingroup$ Let's get back to basics. Aren't a b and c in my question representing 4 dimensions? If yes (please tell me if not), if a[1]==b[1] doesn't that mean that on the second dimension the angel between them is zero? $\endgroup$ – Optimesh Feb 9 at 14:59
  • 2
    $\begingroup$ Angles are not measured dimension by dimension. (Consider the case of two dimensions, where this statement ought to be obvious.) Two non-parallel vectors in any number of dimensions determine a plane (that's one of Euclid's axioms) and the angle between them is measured in that plane. It's the usual concept of angle. $\endgroup$ – whuber Feb 9 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.