3
$\begingroup$

I have a corpus I want to perform an LDA on, however it has very few total words and some words occur extremely often. I want to penalize these words. A tfidf at first seemed intuitive (and I have found literature where this has been used on LSA's). However and LDA expects a bag of words (and those words occur in integers...doesn't really make sense for the word "fun" to occur 2.5 times in a document).

Are there any good pre/post processing techniques to get similar effects as a tfidf while working on an LDA?

$\endgroup$
2
  • $\begingroup$ What is your aim in using latent Dirichlet allocation? Specifically, are you using it solely as an unsupervised technique to explore the corpus, or using the document distributions as features in a supervised learning model? $\endgroup$ Commented Jun 21, 2015 at 19:16
  • $\begingroup$ I have a large corpus that I want to train on and new unseen documents that I want to classify into topics using the result of the LDA's training $\endgroup$
    – sedavidw
    Commented Jun 22, 2015 at 13:02

1 Answer 1

2
$\begingroup$

This paper presents term weighting schemes for LDA, and reports improved precision on an example retrieval task. I've not verified these results, and it doesn't mention an available software implementation, but this may be a good place to start. In brief, it suggests using term weight rather than raw count, and presents a few different weighting schemes (logarithmic and point-wise mutual information).

If you'd like to go further, you may wish to peruse the subsequent papers citing this one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.