Typical training data for LDA is a count matrix $D \in R^{n\times m}$, where $n$ denotes the number of documents in the corpus and $m$ is the number of words of the vocabulary. $D_{ij}$ is the number of occurrences of word $j$ in document $i$. It seems to me the training data $D$ does not contain any structural information, for example, the correlation between different words, etc. I am wondering if there is any paper about utilizing (or recover) these structural information to learn LDA.
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
LDA is a "bag of words" model, so a lot of the "structural" information you're talking about (e.g., the order of the words) is not pertinent to LDA.
With respect to "correlations" between words, LDA does take into account co-occurrences of words within a particular document. For instance, if a document containing one word always contains another word, it's highly likely that LDA will assign both of those words to the same topic. See this answer for more details.
Some people have tried to combine LDA with more sophisticated models that are not based on a bag of words. See, for instance, `lda2vec', a technique that simultaneously learns word embeddings (as in word2vec), and topics (as in LDA).
Another good reference is David Blei's review article. In particular, see the section where he discusses "relaxing the assumptions of LDA".
$\begingroup$
$\endgroup$
What about n-gram bag of words? Maybe that would give a little bit of structural and contextual information. It should be trivial to implement but the computational complexity would explode as expected.
