Using LDA in non-realtime twitter data

Question

I'm trying to understand user characterization from twitter data. How can I infer a user's interests from their status updates? LDA (Latent Dirichlet Allocation) seems to be a suitable approach to topic modeling, based on my reading. I've collected data, removed stop words and punctuation. However, I don't have any training data or example topics. Do I need to specific a topic list for lda? Also, which is the best java library for me?

Answer 1

You might want to look at the Java package jLDADMM.

jLDADMM provides implementations of the LDA topic model and the one-topic-per-document Dirichlet Multinomial Mixture (DMM) model for modeling topics on normal or short texts

Applying topic models, such as LDA, for short texts (e.g. Tweets) is more challenging because of data sparsity and the limited contexts in such texts. One approach is to combine short texts into long pseudo-documents before training LDA. Another approach is to assume that there is only one topic per document like in the DMM topic model.

Answer 2

Part of the beauty of Latent Dirichlet Allocation is that you do not specify the topics a priori. Instead, the topics are discovered in tandem with the topic assignments themselves. In other words, you do not need a specific topic list for LDA.

As for Java libraries, one of the most popular tools which implement LDA is MALLET, which was produced by Andrew McCallum. The quick start includes information on how to get started with topic modeling using the toolkit.

Answer 3

You don't need to specify an a priori list of topics, but you do need to specify the number of topics you'd like the algorithm to find. With a diverse data set like twitter updates, this is going to be a tricky process. I recommend randomly sampling a set of data on which to perform parameter optimization experiments, to see which topic number setting appears to give you the best results, then using the selected parametrization on the remaining data. GraphLab has some nice java-based LDA functionality, but you'll have to do some manual pre-processing with your data to get it into their expected format. Their topic modeling documentation is pretty straightforward, and should give you a nice overview of the typical experimental workflow for topic modeling studies.

Answer 4

Take a look at Vowpal Wabbit software package. It scales even to very big datasets.

From my experience with short documents (not tweets though) it is difficult to make sense of the topics and document clusters that you will find. The clusters of short documents I obtained from my real world data were really complicated. The best results were obtained if some prior information or higher level features are added. Although this was hard, the clusters obtained were much easier to interpret.

For visualization of how the documents cluster after LDA (each document expressed as a distribution over topics), I used SOMOCLU and ESOM. Both are the methods of dimensionality reduction and are useful for visualization purposes.

LATE UPDATE: Working with topic analysis for short texts, it is better to work with term-term representation rather than the standard document-term representation. The extracted topics for my short texts were of better quality (subjective).