Influence of word counts from DTM on LDA with Gibbs Sampling

Question

I'm trying to wrap my head around Topic Modeling based on LDA with Gibbs sampling (Griffiths, Steyvers 2004: Finding Scientific Topics). What struck me when reading some Python implementations like this one or that one, was that it appears that the actual word counts in the input matrix (the document-term-matrix – DTM) are never used in the calculations. When iterating over the words in a document, it is done so by iterating over the non-zero occurrences of words in the vocabulary (for w in word_indices(matrix[m, :]): ... – m denotes a document index) but the actual word count is not used in the calculations for the probability distribution p_z (prob. across topics given a word in a document).

So it appears to me, that for the algorithm it is only important if a word exists in a document and not how often it occurs there. If this was the case, then with a binary DTM that only denotes if a certain word occurs in a document or not, we would achieve similar results as with a conventional DTM containing the number of word occurrences. However, this is not (or should not be) the case and this confuses me. Can someone clarify this? Did I miss something or are the referenced implementations wrong? How and where do the actual word counts in a DTM influence the results?

Answer 1

I guess I found my error in reasoning. I thought that the LDA algorithm operates on the document-term-matrix (DTM) containing the raw word counts. However, in the implementation of the lda python package, I found that the DTM is "expanded" to a binary matrix where the occurrences of words per document are repeated (technically, it's only an array with the repeated indices of the words).

So this DTM with two documents d1 and d2 and a vocabulary consisting only of "foo" and "bar":

   | foo | bar
--------------
d1 |   3 |   2
d2 |   1 |   0

Becomes:

   | foo | foo | foo | bar | bar
--------------------------------
d1 |   1 |   1 |   1 |   1 |   1
d2 |   1 |   0 |   0 |   0 |   0

So LDA treats each occurrence of a word independently, while I initially thought that it uses a weight like the count 3 for a word "foo" and then calculates the prob. distribution based on that weight.

That's the case for LDA using Gibbs sampling at least. Regarding the other approaches, I have no idea.