I have a list of articles, a domain of words/stems and a calculated tf-idf matrix for them. What distance metric should I use when I try to calculate the similarity of two documents?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I don't know much about working with documents, but an interesting approach to documents was taken by Hinton & Salakhutdinov and can be found in this paper (and also in this Google Tech Talk). They used autoencoders to compress documents into low-dimensional, real-valued vectors. The documents appeared to be fairly well clustered in this transformed space, so that I could imagine that even the euclidean metric could give some decent results. Better results can probably be achieved by binarizing the document representations (as described in the talk) and using the Hamming distance.
$\begingroup$
$\endgroup$
Have a look at this paper: Text similarity: an alternative way to search MEDLINE.
They compare the simple cosine similarity with a modified version and also some more complex approaches based on text alignment. The conclusion was that cosine similarity with a small modification performed best, although only slightly better than the standard cosine similarity. Note that this was in a medical context, but that shouldn't matter.
There is also the Okapi BM25 similarity measure which is used quite often and may be worth looking at too.
