Return to Answer

Note that this formulation has a strong requirement: All ranks must be distinct, so even for ties (if two words have the same count in the same document), you need to assign them distinct ranks; I suggest, you use alphanumeric ordering in those cases. And for words that only occur in one document, you place them in the last position(s) in the other document (again using the same alphanumeric ordering).

Note that this formulation has a strong requirement: All ranks must be distinct, so even for ties (if two words have the same count in the same document), you need to assign them distinct ranks; I suggest, you use alphanumeric ordering in those cases. And for words that only occur in one document, you place them in the last position(s) in the other document (again using the same alphanumeric ordering). It should be noted that by re-weighting your counts with TF-IDF before calculating this similarity, you can have far less ties among words with non-zero counts.

Cosine similarity is used most frequently in general, but you should always measure first and make sure that no other similarity would produce better results for your problem, evaluating if you can use some of the more complex measures. Except for Jaccard's set similarity, you can (indeed, might want to) apply some form of TF-IDF reweighing (ff) to your document counts/vectors before using these measures.

Cosine similarity is used most frequently in general, but you should always measure first and make sure that no other similarity would produce better results for your problem, evaluating if you can use some of the more complex measures. Except for Jaccard's set similarity, you can (indeed, might want to) apply some form of TF-IDF reweighing (ff) to your document counts/vectors before using these measures. The TF weighting term might even be Okapi-BM25, and the IDF term replaced with corpus entropy, but in the end has little to do with the original question at hand (BoW similarity measures).

Let me address this by describing the four maybe most common similarity metrics for bags of words and document (count) vectors in general.

Cosine similarity is used most frequently in general, but you should always measure first and make sure that no other similarity would produce better results for your problem, evaluating if you can use some of the more complex measures.

Note that the first measure, Jaccard similarity, suffers from a significant downside: It is the most heavily affected by length differences among documents, and does not take word frequencies into account, either. All the other measures are based on word frequencies (counts), and you can (or rather, should) adjust the vectors by normalizing your vectors to unit length. That can happen in addition to any proper TF-IDF setup where you re-scale your vectors. But even then, shorter documents will have more sparse counts (i.e., more zeros in their vectors) compared to longer documents, for obvious reasons. (One way around that sparsity difference could be to only include words above some minimum cutoff, using a dynamic cutoff value that increases with document length.)

Let me address this by describing the four maybe most common similarity metrics for bags of words and document (count) vectors in general, that is comparing collections of discrete variables.

Cosine similarity is used most frequently in general, but you should always measure first and make sure that no other similarity would produce better results for your problem, evaluating if you can use some of the more complex measures. Except for Jaccard's set similarity, you can (indeed, might want to) apply some form of TF-IDF reweighing (ff) to your document counts/vectors before using these measures.

Note that the first measure, Jaccard similarity, suffers from a significant downside: It is the most heavily affected by length differences among documents, and does not take word frequencies into account, either. For Cosine similarity and the $\chi^2$-distance you can (or rather, should) adjust the vectors by normalizing your vectors to unit length (i.e., in addition to TF-IDF re-weighting). But even then, shorter documents will have more sparse counts (i.e., more zeros in their vectors) compared to longer documents, for obvious reasons. (One way around that sparsity difference could be to only include words above some minimum cutoff, using a dynamic cutoff value that increases with document length.)