I have been using Latent Dirichlet Analysis for a while but I am a bit confounded as to it's practical ability to compare two documents. It is of course ideal for classification when you want to see in what category a certain document or word belongs, but in content comparison I am at my wits end.
An inferred document yields a distribution over n topics, summing to one.
Using Kolmogorov-Smirnov was recommended somewhere, but it has the annoying property of giving two identical distributions a very high score, even if they do not make sense. Two words not in the dictionary will give a perfect match, which is of course absurd.
The normalized dot product actually works very well because it punishes flat distributions (trivial documents), but I thought there would be a better one.
Any suggestion is appreciated.