On the spark implementation of word2vec, when the number of iterations or data partitions are greater than one, for some reason, the cosine similarity is greater than 1.

In my knowledge, cosine similarity should always be about $-1 < \cos\theta < 1$. Does anyone know why?

  • $\begingroup$ Not sure about MlLib, but it happens on Scipy too. They use a different formula. Check this answer. So, check out whether MlLib uses a different formula too. $\endgroup$ – Dawny33 Oct 27 '15 at 5:07
  • $\begingroup$ Perhaps the vectors are not normalized? $\endgroup$ – Vladislavs Dovgalecs Oct 27 '15 at 5:37

The Spark documentation for this kind of thing doesn't seem very thorough, so I looked at the source. There's a comment here saying

// Need not divide with the norm of the given vector since it is constant.

This seems consistent with the following code.

So, it seems that findSynonyms doesn't actually return cosine distances, but rather cosine distances times the norm of the query vector. The ordering and relative values are consistent with the true cosine distance, but the actual values are all scaled.

Not sure why the number of iterations or data partitions should have any bearing on this.

| cite | improve this answer | |
  • $\begingroup$ Hi . Thanks for response. I am getting the same error and thinking why is that happening. Normalization will not have any effect on the cosine values. You are saying the output of spark findSynonyms is not a cosine similarity but cosine similarity*norm(input query). Here input query you mean is the word whose synonyms we are trying to find? If yes, then if I do w2vecmodel.transform(input_word) and then get the norm of that vector and dividing the output of findSynonmys by that it should be the cosine distance then? $\endgroup$ – Baktaawar Dec 29 '16 at 21:36
  • $\begingroup$ @Baktaawar Yes, that seems to be the case based on my reading of the code as someone who's never used Spark, but you should run some checks yourself before relying on that. $\endgroup$ – Dougal Dec 29 '16 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.