0
$\begingroup$

On the spark implementation of word2vec, when the number of iterations or data partitions are greater than one, for some reason, the cosine similarity is greater than 1.

In my knowledge, cosine similarity should always be about $-1 < \cos\theta < 1$. Does anyone know why?

Improve this question
$\endgroup$
2
  • $\begingroup$ Not sure about MlLib, but it happens on Scipy too. They use a different formula. Check this answer. So, check out whether MlLib uses a different formula too. $\endgroup$
    – Dawny33
    Oct 27, 2015 at 5:07
  • $\begingroup$ Perhaps the vectors are not normalized? $\endgroup$
    – Vladislavs Dovgalecs
    Oct 27, 2015 at 5:37

1 Answer 1

Reset to default
4
$\begingroup$

The Spark documentation for this kind of thing doesn't seem very thorough, so I looked at the source. There's a comment here saying

// Need not divide with the norm of the given vector since it is constant.

This seems consistent with the following code.

So, it seems that findSynonyms doesn't actually return cosine distances, but rather cosine distances times the norm of the query vector. The ordering and relative values are consistent with the true cosine distance, but the actual values are all scaled.

Not sure why the number of iterations or data partitions should have any bearing on this.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi . Thanks for response. I am getting the same error and thinking why is that happening. Normalization will not have any effect on the cosine values. You are saying the output of spark findSynonyms is not a cosine similarity but cosine similarity*norm(input query). Here input query you mean is the word whose synonyms we are trying to find? If yes, then if I do w2vecmodel.transform(input_word) and then get the norm of that vector and dividing the output of findSynonmys by that it should be the cosine distance then? $\endgroup$
    – Baktaawar
    Dec 29, 2016 at 21:36
  • $\begingroup$ @Baktaawar Yes, that seems to be the case based on my reading of the code as someone who's never used Spark, but you should run some checks yourself before relying on that. $\endgroup$
    – Danica
    Dec 29, 2016 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.