I understand what cosine similarity is and how to calculate it, specifically in the context of text mining (i.e. comparing tf-idf document vectors to find similar documents). What I'm looking for is some better intuition for interpreting the results/similarity scores I come up with.

My question: If I have a cosine similarity of less than 0.707 (i.e. an angle greater than 45 degrees), is is fair to say that those respective documents/vectors are more "different" (less "similar") since the angle between them is more orthogonal? My initial thought was 'yes,' but in practice for me so far it doesn't seem like that's the right way to read into the numbers.

  • 4
    $\begingroup$ Cosine similarity is like Pearson correlation, only done on not centered data. For binary data, it is known also as Ochiai coefficient. Yes, cosine similarity bw two vectors is, like r, is the cosine of the angle between them. $\endgroup$ – ttnphns Jan 17 '17 at 20:59
  • $\begingroup$ You are asking specifically about using it in text mining. I'm not text analyst, I just suppose the compared documents are formed as binary data, yes? Then probably the meaning is the meaning of Ochiai? (see its formula). $\endgroup$ – ttnphns Jan 17 '17 at 21:04

Cosine similarity is computed all the text documents after pre processing , like removals of stop words, stemming and applying Term frequency. Let us say A,B,C,D are four documents, I need to find out the similarity I can apply cosine similarity after going through all the preprocessing and calculating the weigths. Finally determine which are similarity based on the Angel, Now I would like to see the key tokens which contribution, (like man of the match in Cricket) and what semantic it is occurring weather is object or thing or Noun or Verb and predict the tokens which is relevant , For instance ,I live in Java and place is good. Here the context is JAVA ia Island , not bike nor oops. These way CS(cosine similarity is applied and predictive approach is dtermeined)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.