I was wondering if anyone could provide some intuition about when the "documents" in LDA are too small for it to provide any benefit. For instance, I have seen papers where people have you used context windows of 5 words as 'documents" which I find to not make too much sense as the model is only taking words into account (no structure). Clearly at some point clustering that takes more than word counts must be used instead of LDA. By this I mean taking word similarity into account or structural info. Is there any intuition about the nature of this relationship?


1 Answer 1


The biggest issue with LDA is that it has no concept of correlation between topics. Blei et al create the Correlated Topic Model to work and getting around that limitation in particular. But this is just one step.

Your issue I think (if I understand correctly) stems from the fact that we are doing "clustering" on human text, but we have thrown away all the sentance structure. Yet somehow we continue to get very good results with LDA anyway. The more "formal" modeling of sentence structure and individual word correlations used to be the preferred way, but it was very difficult. No one has really developed any good algorithms for consistently deriving the structure of arbitrary "valid" sentence (where I mean that the sentence is not human produced gibberish). Chomsky is a proponent of developing such models that take into account everything that is going on.

Ultimately though, after people tried the more "stupid" ways (BoW with bi/trigrams +LDA and other simple classifiers) the performance of our systems went up, especially if you threw a lot of data at it. Not only that, but they are easier to build and train. So they became the norm.

So to answer your question more succinctly. LDA almost never fits perfectly with what we are doing. Neither does SVMs or Decision Trees, or almost any model really. They are all approximations that we make to simplify our models so that we can actually build, train, and use them. Therefore we try to pick the model that is "least wrong" for our goals, since using all of the true structure is too hard / difficult for us so far.

  • $\begingroup$ Thanks. The thing is structure is less important for topic modeling when you have lots of words. When your "documents" are small, LDA should decrease in effectiveness. For instance documents of one word cannot be used with LDA. Does LDA make sense for documents of say 2 words? or 5 words? $\endgroup$
    – user671931
    Dec 8, 2013 at 18:41
  • $\begingroup$ I think you are thinking of it in a weird way. The real question is "can the topics discovered by LDA occur in datasets with small vocabularies". Or conversely, "are small vocabulary documents likely to produce topics that LDA can handle". LDA as an algorithm certain works on small documents, but the topics/document types that occur in such small vocabularies might be very different from what LDA is normally used for. $\endgroup$ Dec 9, 2013 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.