1
$\begingroup$

I have been using Latent Dirichlet Analysis for a while but I am a bit confounded as to it's practical ability to compare two documents. It is of course ideal for classification when you want to see in what category a certain document or word belongs, but in content comparison I am at my wits end.

An inferred document yields a distribution over n topics, summing to one.

Using Kolmogorov-Smirnov was recommended somewhere, but it has the annoying property of giving two identical distributions a very high score, even if they do not make sense. Two words not in the dictionary will give a perfect match, which is of course absurd.

The normalized dot product actually works very well because it punishes flat distributions (trivial documents), but I thought there would be a better one.

Any suggestion is appreciated.

$\endgroup$
0
$\begingroup$

There's numerous approaches to this. For a pair of documents, the Kullback Leibler divergence of their respective topic distributions is probably the most common and straight-forward.

See page 12 of Probabilistic Topic Models by Styvers and Griffiths for a discussion of the KL divergence approach, as well as several others.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ As far as I can see, my particular problem of TRIVIAL identical distributions is not solved by Shannen-Jensen / Kullback-Leibler. Look!! Sum_i[ ln(Pi/Qi) * Pi] means that all precisely similar ps and qs make it ln(1) * whatever = 0. It measures closeness only too good, because it assumes that information in itself has no variance-properties. Very weird since Shannon was an information theorist. Or am I missing something big? Stats appears to be too tough for me.. $\endgroup$ – user19795 Jan 21 '13 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.