# Comparing cosine similarities for tf-idf vectors for documents with different length

I'm computing cosine similarities between 2 vectors.

These vectors are information retrieval query and document representations respectively.

They have been computed using tf-idf weights.

Since my documents have different length, tf-idf weights are theoretically unbounded.

The question is: is cosine similarity still a valid measure? Can I compare several cosine similarities for each doc?

According to Wikipedia's article of tf-idf:

The term count in the given document is simply the number of times a given term appears in that document. This count is usually normalized to prevent a bias towards longer documents (which may have a higher term count regardless of the actual importance of that term in the document) to give a measure of the importance of the term t within the particular document d

So, normalize the frequency of a term t by the length of the document d in which it occurs. Then you can compute cosine similarity between your tf-idf vectors.

• I'm trying to implement tf-idf in MATLAB from scratch. I have written functions to calculate tf and idf vectors for any document. The next step is to get cosine distance between every pair of tf-idf vectors representing each document. The problem is that since there are different number of unique words in every document, the tf and idf vectors I calculate for every document have different lengths. In such a case, how do I calculate the cosine similarity? Any help is appreciated. – Shashwat Siddhant Dec 3 '19 at 19:33
• This was 8 years ago, so it's a bit fuzzy, but I guess one solution would be to insert the same words into both vectors, setting their frequencies to 0 when they don't appear in a document. For instance, if the word APPLE appears in Document A, but not in Document B, insert it into both vectors, setting it to 0 for Document B's vector. Again, 8 years ago, so there may be far better ways, but this is one way to tackle it. – ctc Dec 3 '19 at 22:13
• That makes sense. Thank you so much for your response! – Shashwat Siddhant Dec 4 '19 at 11:44

The cosine similarity is still a valid measure. Actually, this is the rule that tf-idf weights have different lengths for different documents, simply because they do not use exactly the same words. Notice that a missing word in a tf-idf vector is actually a word with a frequency of 0.

So you elongate both vectors to the same length by adding and couple of 0's and youb compute the cosine similarity.