I'm trying to implement Latent Dirichlet Allocation (LDA) on a bigram language model. This is described in Topic Modeling: Beyond Bag-of-Words by Hanna Wallach et al.

I'm trying to easily implement this idea using the current LDA packages (for example python lda.lda).

Here is the idea I thought of:

Normally we introduce lda.fit(X) where X is a DxN bag of words matrix (D is number of documents, N is number of words in document, and each xij is the count for word j in document i).

Instead we could introduce lda.fit(Y) where Y is a DxL bag of unigram and bigram words matrix (D is number of documents, L is addition of number of words and number of bigram options in document. Each yij is the count for word/bi-word j in document i).

Will the rest of the algorithm work the same, and output a list of topics with a probability distribution of unigram and bigram words?

Do you think this will work? Do you have any other idea for implementing bigram LDA?

  • $\begingroup$ If I got it right, I believe it could work because of what I saw in Biterm. Luckily, the original paper can provide you with some insights. $\endgroup$ – Felipe Martins Melo Apr 30 '15 at 13:48
  • $\begingroup$ There's code for N-gram topic models in the Mallet toolkit that you might find useful for thinking about data structures, albeit in a slightly different model. $\endgroup$ – conjugateprior Apr 30 '15 at 15:13
  • $\begingroup$ I agree with your ideas.Does bigram LDA fit short text more due to time complexity? $\endgroup$ – user79373 Jun 9 '15 at 20:13
  • $\begingroup$ I don't think this follows the definition of Wallach's publication. It is still a bag of bigrams and unigrams, where in his model, all word are connected in a markov chain fashion. $\endgroup$ – Tamaki Sakura Apr 6 '16 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.