I was wondering if anyone could provide some intuition about when the "documents" in LDA are too small for it to provide any benefit. For instance, I have seen papers where people have you used context windows of 5 words as 'documents" which I find to not make too much sense as the model is only taking words into account (no structure). Clearly at some point clustering that takes more than word counts must be used instead of LDA. By this I mean taking word similarity into account or structural info. Is there any intuition about the nature of this relationship?
The biggest issue with LDA is that it has no concept of correlation between topics. Blei et al create the Correlated Topic Model to work and getting around that limitation in particular. But this is just one step.
Your issue I think (if I understand correctly) stems from the fact that we are doing "clustering" on human text, but we have thrown away all the sentance structure. Yet somehow we continue to get very good results with LDA anyway. The more "formal" modeling of sentence structure and individual word correlations used to be the preferred way, but it was very difficult. No one has really developed any good algorithms for consistently deriving the structure of arbitrary "valid" sentence (where I mean that the sentence is not human produced gibberish). Chomsky is a proponent of developing such models that take into account everything that is going on.
Ultimately though, after people tried the more "stupid" ways (BoW with bi/trigrams +LDA and other simple classifiers) the performance of our systems went up, especially if you threw a lot of data at it. Not only that, but they are easier to build and train. So they became the norm.
So to answer your question more succinctly. LDA almost never fits perfectly with what we are doing. Neither does SVMs or Decision Trees, or almost any model really. They are all approximations that we make to simplify our models so that we can actually build, train, and use them. Therefore we try to pick the model that is "least wrong" for our goals, since using all of the true structure is too hard / difficult for us so far.