1
$\begingroup$

I want to employ Latent Dirichlet Allocation (LDA) for topic modeling and I'm trying out the implementation from scikit-learn for that.

Running the example (which uses messages from newsgroups as documents) from scikit's documentation works just fine and delivers reasonable results, but when I'm trying out any other data set, I get some very strange results. For example, when I'm using the 18 documents from the Gutenberg corpus provided with NLTK and fit a topic model with 30 topics, many of those topics are essentially the same and have all-equal weights (each weight equal to the topic_word_prior parameter), see for example the following output:

Topic #0: zoological (0.01) fathoms (0.01) fatigued (0.01) fatigues (0.01) fatiguing (0.01)
Topic #1: mr (1081.61) emma (866.01) miss (506.94) mrs (445.56) jane (301.83)
Topic #2: zoological (0.01) fathoms (0.01) fatigued (0.01) fatigues (0.01) fatiguing (0.01)
Topic #3: thee (82.64) thou (70.0) thy (66.66) father (56.45) mother (55.27)
Topic #4: anne (498.74) captain (303.01) lady (173.96) mr (172.07) charles (166.21)
Topic #5: zoological (0.01) fathoms (0.01) fatigued (0.01) fatigues (0.01) fatiguing (0.01)
Topic #6: zoological (0.01) fathoms (0.01) fatigued (0.01) fatigues (0.01) fatiguing (0.01)
Topic #7: zoological (0.01) fathoms (0.01) fatigued (0.01) fatigues (0.01) fatiguing (0.01)
...

Code:

import nltk
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        message = "Topic #%d: " % topic_idx
        message += " ".join([feature_names[i] + " (" + str(round(topic[i], 2)) + ")"
                             for i in topic.argsort()[:-n_top_words - 1:-1]])
        print(message)


all_docs_en = {f_id: nltk.corpus.gutenberg.raw(f_id)
               for f_id in nltk.corpus.gutenberg.fileids()}

data_samples = all_docs_en.values()

tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                stop_words='english')
tf = tf_vectorizer.fit_transform(data_samples)

lda = LatentDirichletAllocation(n_components=30,
                                learning_method='batch',
                                n_jobs=-1,  # all CPUs
                                verbose=1,
                                evaluate_every=10,
                                max_iter=1000,
                                doc_topic_prior=0.1,
                                topic_word_prior=0.01,
                                random_state=1)

lda.fit(tf)
tf_feature_names = tf_vectorizer.get_feature_names()
print_top_words(lda, tf_feature_names, 5)

Changing some parameters didn't help, but using a different Python implementation (e.g. the lda package) results in reasonable topics. However, I need to use scikit-learn because it runs on multiple processors, hence I'm a bit stuck here. I'm wondering if there might be a bug in the implementation or if I'm missing something.

I should add that I'm using Python 2.7 and scikit-learn 0.19.0.

$\endgroup$
3
  • $\begingroup$ Have you tried if with different random_state parameters, and if so did you observe the same result? $\endgroup$ Sep 15, 2017 at 18:44
  • $\begingroup$ Same. It doesn't depend on the random_state. However, I noticed similar, but not as extreme, behavior when using Gensim, which also uses the Variational Bayes approach. Could it be the Variational Bayes approach is problematic when having only a small number of (though quite big) documents? Or do I have to adjust the parameters somehow in this case? $\endgroup$ Sep 18, 2017 at 8:01
  • $\begingroup$ Rereading, I see that you've set num_topics to less than the size of the corpus. I'd give it a try with num_topics << N. $\endgroup$ Sep 18, 2017 at 15:41

1 Answer 1

1
$\begingroup$

In a small discussion in the scikit-learn mailinglist, we came to the conclusion that the number of documents is indeed to few for the algorithm to learn informative topics from it. However, this is really a limitation of the Online Variational Bayes (Hoffman et al) approach which is implemented in scikit-learn for LDA (the same approach is also used in Gensim). If I split the documents into "sub-documents" with, say, 5 paragraphs each (giving ~10k of such sub-documents), the algorithm is able to find informative topics.

Gibbs sampling based implementations like lda, on the other hand, handle scenarios with a small number of (large) documents better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.