Comparing cosine similarities for tf-idf vectors for documents with different length

Question

I'm computing cosine similarities between 2 vectors.

These vectors are information retrieval query and document representations respectively.

They have been computed using tf-idf weights.

Since my documents have different length, tf-idf weights are theoretically unbounded.

The question is: is cosine similarity still a valid measure? Can I compare several cosine similarities for each doc?

Answer 1

According to Wikipedia's article of tf-idf:

The term count in the given document is simply the number of times a given term appears in that document. This count is usually normalized to prevent a bias towards longer documents (which may have a higher term count regardless of the actual importance of that term in the document) to give a measure of the importance of the term t within the particular document d

So, normalize the frequency of a term t by the length of the document d in which it occurs. Then you can compute cosine similarity between your tf-idf vectors.

Answer 2

The cosine similarity is still a valid measure. Actually, this is the rule that tf-idf weights have different lengths for different documents, simply because they do not use exactly the same words. Notice that a missing word in a tf-idf vector is actually a word with a frequency of 0.

So you elongate both vectors to the same length by adding and couple of 0's and youb compute the cosine similarity.