1
$\begingroup$

I am assessing the similarity between documents represented as vectors of tf-idf values. I know that the cosine similarity is a well-defined and commonly used measure in information retrieval.

However I am thinking how to deal with the case where 2 documents are "linearly dependent".

Let's consider three documents:

$d_1 = [1,1,3,4] $

$d_2 = [1,1,2,4] $

$d_3 = [10,10,30,40] $

At first sight, $d_1$ and $d_2$ should be more similar than $d_1$ and $d_3$.

However, using the cosine similarity:

import numpy as np
def cos_sim(d1,d2):
    return np.dot(d1,d2)/ (np.sqrt(np.dot(d1,d1))* np.sqrt(np.dot(d2,d2)))

I got:

cos_sim(d1,d2) = 0.98473192783466179

cos_sim(d1,d3) = 1.0

How should we deal with this case?

$\endgroup$
  • 1
    $\begingroup$ If you want to keep the meaning of similar in your sentence starting "At first sight ..." you probably want to use something other than cosine similarity. $\endgroup$ – mdewey Mar 12 '17 at 12:20
  • $\begingroup$ Could you please develop your answer? $\endgroup$ – Ale Mar 12 '17 at 12:22
  • 2
    $\begingroup$ I do not agree. cosine similarity is higher when vectors are highly correlated (aligned). d3=10*d1 (perfect correlation) whereas d2 is not. $\endgroup$ – jpmuc Mar 12 '17 at 12:25
  • $\begingroup$ One of the reasons cosine similarity is used for comparing documents is that it's invariant to the actual number of times each term is used; only the relative frequencies matter. This way a long document with many words can be similar to a short document with fewer words but similar frequencies. If this isn't something you want, it would be better to choose a different metric. $\endgroup$ – user20160 Mar 12 '17 at 13:48
2
$\begingroup$

Recent Article have shown that "ts-ss" is better than cosine or euclidean

because Cosine, Euclidean drawbacks

Reference paper is "A Hybrid Geometric Approach for Measuring Similarity Level Among Documents and Document Clustering"

If you want to see a summary of the paper, please refer to the github below.

https://github.com/taki0112/Vector_Similarity

thank you

$\endgroup$
2
$\begingroup$

A short answer is that cosine similarity works best when there are a great many (and likely sparsely populated) features to choose from. Under these conditions, Euclidean methods tail off in terms of their sensitivity. Likewise, cosine similarity is less optimal under spaces of lower dimension.

$\endgroup$
  • $\begingroup$ I understand why Euclidean methods lose sensitivity when you add more and more features, as the incremental "power" of each additional feature decays. However, could you elaborate more on why cosine similarity works well with sparsely many populated features? Why specifically sparse? $\endgroup$ – Yu Chen Jan 8 at 17:13
  • $\begingroup$ I might have mislead you a little with that line - I think of cosine similarity as analogous to measuring the distances between stars in the night-sky - because space is so sparsely populated, it’s a fairly successful heuristic to make angular distance a proxy for true distance. If space were more densely populated, that assumption would be less helpful. Multi-dimensional spaces tend to be even more sparse, so the success of the heuristic is amplified. $\endgroup$ – Thomas Kimber Jan 13 at 10:41
  • $\begingroup$ Got it. Thanks for the explanation. I also realized a little later that cosine similarity is a dot product in its numerator, so it only will take into account the elements in common between two vectors, which is useful for a sparse feature space where most elements will be 0s. $\endgroup$ – Yu Chen Jan 13 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.