6
$\begingroup$

I'm computing cosine similarities between 2 vectors.

These vectors are information retrieval query and document representations respectively.

They have been computed using tf-idf weights.

Since my documents have different length, tf-idf weights are theoretically unbounded.

The question is: is cosine similarity still a valid measure? Can I compare several cosine similarities for each doc?

$\endgroup$
2
$\begingroup$

According to Wikipedia's article of tf-idf:

The term count in the given document is simply the number of times a given term appears in that document. This count is usually normalized to prevent a bias towards longer documents (which may have a higher term count regardless of the actual importance of that term in the document) to give a measure of the importance of the term t within the particular document d

So, normalize the frequency of a term t by the length of the document d in which it occurs. Then you can compute cosine similarity between your tf-idf vectors.

$\endgroup$
3
  • $\begingroup$ I'm trying to implement tf-idf in MATLAB from scratch. I have written functions to calculate tf and idf vectors for any document. The next step is to get cosine distance between every pair of tf-idf vectors representing each document. The problem is that since there are different number of unique words in every document, the tf and idf vectors I calculate for every document have different lengths. In such a case, how do I calculate the cosine similarity? Any help is appreciated. $\endgroup$ – Shashwat Siddhant Dec 3 '19 at 19:33
  • 1
    $\begingroup$ This was 8 years ago, so it's a bit fuzzy, but I guess one solution would be to insert the same words into both vectors, setting their frequencies to 0 when they don't appear in a document. For instance, if the word APPLE appears in Document A, but not in Document B, insert it into both vectors, setting it to 0 for Document B's vector. Again, 8 years ago, so there may be far better ways, but this is one way to tackle it. $\endgroup$ – ctc Dec 3 '19 at 22:13
  • $\begingroup$ That makes sense. Thank you so much for your response! $\endgroup$ – Shashwat Siddhant Dec 4 '19 at 11:44
2
$\begingroup$

The cosine similarity is still a valid measure. Actually, this is the rule that tf-idf weights have different lengths for different documents, simply because they do not use exactly the same words. Notice that a missing word in a tf-idf vector is actually a word with a frequency of 0.

So you elongate both vectors to the same length by adding and couple of 0's and youb compute the cosine similarity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.