Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm computing cosine similarities between 2 vectors.

These vectors are information retrieval query and document representations respectively.

They have been computed using tf-idf weights.

Since my documents have different length, tf-idf weights are theoretically unbounded.

The question is: is cosine similarity still a valid measure? Can I compare several cosine similarities for each doc?

share|improve this question
add comment

2 Answers 2

According to Wikipedia's article of tf-idf:

The term count in the given document is simply the number of times a given term appears in that document. This count is usually normalized to prevent a bias towards longer documents (which may have a higher term count regardless of the actual importance of that term in the document) to give a measure of the importance of the term t within the particular document d

So, normalize the frequency of a term t by the length of the document d in which it occurs. Then you can compute cosine similarity between your tf-idf vectors.

share|improve this answer
add comment

The cosine similarity is still a valid measure. Actually, this is the rule that tf-idf weights have different lengths for different documents, simply because they do not use exactly the same words. Notice that a missing word in a tf-idf vector is actually a word with a frequency of 0.

So you elongate both vectors to the same length by adding and couple of 0's and youb compute the cosine similarity.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.