I have a bunch of text documents of varying lengths (100k words to just thousands). I want to compare similarities of these vectors, specifically, cosine similarities.
While I understand that cosine similarity, unlike euclidean distances, for example, reduce the effects of the initial document length, the
similarity measure is still to some degree a function of the initial document length, since larger texts contain more terms and by definition are more similar than smaller ones.
Is there a way to perform some kind of unit vector normalisation by the initial document length, so that vectors from larger documents are somehow pulled down when computing cosine similarities?