I am working on a dataset which are rather unusual in the following ways:

  1. It doesn't have just natural language text, it has text like user name, even code snippets etc.
  2. Unusually large vocabulary (unique tokens) size (2M) for a 750K set of documents and about 19M tokens.

All the aspects of the dataset are important and have to be included in the training i.e. the usernames, code snippets etc.

I trained an Latent Dirichet Allocation (LDA) after tokenization, removal of stop words and stemming. Training set size is 720K which about 16M tokens. I trained for 200 and 300 topics and 50 and 100 passes over training data.

I was testing on the test set to see a distribution of first 5 most probable topics of each document in the test set.

What I found was that it is falling the Zip's law for both 200 and 300 number of topics.

Can someone explain why this is happening? Less training or more training or what could be reason?

Attached is the distribution of 200(orange) topics and 300(blue) topics. (Sorry about the wrong title.) The graphs are plotted by extracting top-5 topics of each document and then counting the value for each topic i.e. topic-frequency in test set and plotting the frequency in decreasing order.

enter image description here

enter image description here

  • $\begingroup$ It might help to look at what that topic contains compared to the other topics. Also, is the same effect observed in the training set? $\endgroup$
    – e2crawfo
    Commented Oct 9, 2016 at 17:11
  • $\begingroup$ I am trying to visualize using PyLdaVis. I will see what I can do with respect to training set. $\endgroup$
    – silent_dev
    Commented Oct 9, 2016 at 17:14
  • $\begingroup$ I don't follow your visualization: These plot the count of times each topic appears in a document's 5 most probable topics, is that right? $\endgroup$ Commented Oct 9, 2016 at 20:21
  • $\begingroup$ @SeanEaster : You are almost right. The plot shows topics that appear in a document's top 5 most probable topics. $\endgroup$
    – silent_dev
    Commented Oct 9, 2016 at 20:38
  • 2
    $\begingroup$ LDA assigns each document to all the topics, but each with a different proportion. E.g. document is 10% topic 1, 20% topic 2, etc... Are you adding up these proportions or just the number of times each topic has the highest proportion? Doing the former might get you a more even distribution. Also, have you looked at the words with the highest probability in the topic that shows up the most? If it contains words like "if" "while" etc... (words that show up a lot in code), you might need to expand your list of stop words to fit the particular context. $\endgroup$ Commented Oct 11, 2016 at 14:04

2 Answers 2


My first bet would be that the function words in a corpus of source code differ vastly from those of standard stop lists, and that your model's first topic is indeed capturing standard programming fare: if, int, new, while, etc.

Besides building a custom stop list—seeing which words have high probability under the most frequently assigned topics is a good place to start—you might consider fitting a hierarchal topic model, first described in this paper and in more detail in this one. From the first:

In our approach, each node in the hierarchy is associated with a topic, where a topic is a distribution across words. A document is generated by choosing a path from the root to a leaf, repeatedly sampling topics along that path, and sampling the words from the selected topics. Thus the organization of topics into a hierarchy aims to capture the breadth of usage of topics across the corpus, reflecting underlying syntactic and semantic notions of generality and specificity.

Meaning, using this model, all documents start at the root node, which will include the most common words in the corpus. (See the paper for examples.) This lets you avoid determining a list of stop words manually:

The model has nicely captured the function words without using an auxiliary list, a nuisance that most practical applications of language models require. At the next level, it separated the words pertaining to neuroscience abstracts and machine learning abstracts. Finally, it delineated several important subtopics within the two fields. These results suggest that hLDA can be an effective tool in text applications.

Implementation here. (Unaware of one in Python.)

  • $\begingroup$ Would you say this is an equivalent or better implementation of hLDA - radimrehurek.com/gensim/models/hdpmodel.html ? $\endgroup$
    – silent_dev
    Commented Oct 12, 2016 at 3:09
  • $\begingroup$ Says it was adapted from Chang's original code, so I would think they're comparable. (Will try and take a closer look as time allows.) gensim is in reasonably wide use, which brings confidence. $\endgroup$ Commented Oct 12, 2016 at 4:26
  • $\begingroup$ @user3667569 Sorry, realized I'd confused the implementations: The ones I gave were for hierarchal Dirichlet process topic models, a different model that allows for infinite topics but without hierarchy. hLDA, which has hierarchies, has no Python implementation that I can find. (Linked one in edits is in c.) $\endgroup$ Commented Oct 15, 2016 at 21:57

I found the problem. As indicated by several comments and the answer, it seemed LDA was doing fine and it was. The only problem was the data was too much for LDA and too noisy. I was training on 750K+ docs and everything was noisy. Upon reducing the data to 30k relevant docs, I was able to achieve much better results.

  • $\begingroup$ What techniques you used to reduce the number of documents ? $\endgroup$
    – Siddharth
    Commented Oct 13, 2018 at 4:20
  • $\begingroup$ I randomly sampled to 30000 documents. $\endgroup$
    – silent_dev
    Commented Oct 14, 2018 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.