I am using the Weka API for my research about document classification. I wish to apply Latent Dirichelet Allocation on my dataset followed by using a classifier in Weka. However, it is not so clear to me how I can perform LDA on my raw textual data and represent it in a way I can feed it to the classifiers in Weka since the format needs to be in ARFF right?

Could someone explain a bit more how I can achieve something like this? Someone who already had experience doing this?


Weka can use data from multiple formats, for example the simplest is csv input.

Now, first you would need some library for doing LDA, I would suggest you to try R for that there is a fantastic lda package which does all you need. There are also java packages, it really doesn't matter what you choose the process is pretty much the same

The main thing you need to do is pre-process the text. For example, you filter out short words (e.g., the, a, on, of) and for example convert everything to lowercase. You do that for all the documents in your corpus.

Then you need to create a specific object called Document-Term Matrix. The rows in the matrix are individual documents, and the columns are the words. The values in each cell are 1s or 0s indicating whether a word appears in a given document.

You make this matrix by first finding all unique words from all the documents and create a long list of them. Then, for each document you make a list of 1s and 0s which indicates whether a word at particular position appears in that document. Then you either make a nested list (in Java it would be List>) or matrix (in Java it would be Integer[][]) or some similar object depending on a particular library you're using.

The next thing you need to define is the number of topics you're interested in extracting. Probably you'll try different values to see which one looks the best for your problem. There are also some more formal ways of deciding the number of topics, but this is less important for you at the moment.

After this, you just invoke the LDA algorithm where you provide this document term matrix and the desired number of topics and the rest is magic :)

As a result, you'll be given for each document the most likely topic which you can use as a numeric feature in Weka as any other classification feature.

  • $\begingroup$ Thanks for your input, however I wonder if this approach is scalable? In my case I have a dataset with +/- 100,000 documents, after preprocessing (stopword removal, lowercasing, and even stemming!) I have ~300,000 features (or as Weka calls them, attributes). Making such Document-Term Matrix means 100,000 x 300,000 entries. Followed by some easy calculations, one can see right away this amount does not fit in a normal, even decent desktop's/laptop's memory. $\endgroup$ – GreatEyes Apr 15 '14 at 20:30
  • $\begingroup$ It does not mean that you need to store it in that way. If you have that many documents and terms, then you can store them in a sparse way. For example, even though each document has 300 000 columns, the 99% of them are 0s. So instead of storing full vector (e.g.. [0,1,0,0,0,1,0]) you can store just [1,5] which means that at the indices and 1 and 5 there are 1s and the rest are zeros. I'm pretty sure R can use those sparse matrices, you can check how to do that in Java, there must be a way :) $\endgroup$ – Vitomir Kovanovic Apr 16 '14 at 2:10

See Blie's original LDA paper for how to set something like that up. He uses the topic assignments from LDA as the new features per data point.

  • 1
    $\begingroup$ Thanks for this, however don't you think this is a drastic approach that leads to quite some loss in data? For example, I have tested and implementation LDA on my dataset and in most cases LDA agrees that each feature in a document is assigned to one topic. So if I would replace all features with just this one topic, is the idea of LDA - each document is a mixture of topics - gone? $\endgroup$ – GreatEyes Apr 15 '14 at 20:24
  • $\begingroup$ That sounds like either your data or your LDA implementation is very bad. You can see the result's in the paper that introduced LDA that it does work and retains discriminatory information despite learning without labels. $\endgroup$ – Raff.Edward Apr 16 '14 at 0:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.