0
$\begingroup$

I have a bunch of text documents of varying lengths (100k words to just thousands). I want to compare similarities of these vectors, specifically, cosine similarities.

While I understand that cosine similarity, unlike euclidean distances, for example, reduce the effects of the initial document length, the similarity measure is still to some degree a function of the initial document length, since larger texts contain more terms and by definition are more similar than smaller ones.

Is there a way to perform some kind of unit vector normalisation by the initial document length, so that vectors from larger documents are somehow pulled down when computing cosine similarities?

$\endgroup$
0
$\begingroup$

larger texts contain more terms and by definition are more similar than smaller ones.

As you remarked, in cosine similarity calculation one scales dot product with document vector's length. This makes each word in longer document less relevant.

In addition to that, most applications of Bag-of-Words models actually use TF-IDF in addition to counts. That also makes problem of longer documents less severe, as the common words' weights are multiplied by (some function of, for example logarithm of) inverse document frequency.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.