Running Latent Dirichlet Allocation (LDA) on word counts

Question

I have difficulties understanding the VB implementation lda-c. In particular, the method expects as input a bag-of-words representation of documents, where distinct words appearing in a document are mapped to the number of their occurence in that document. The generative model, however, seems to be specified in terms of a sequence of words per document denoted by $w_{d, n} \in \{1,\ldots,V\}$ for the $n$-th word taken from our vocabulary of size $V$ in the $d$-th document. Each word is assigned a latent topic variable $z_{d,n} \in \{1, \ldots, K\}$. Given $z_{d,n}$ and word distributions $\beta_k$ per topic $k$, the likelihood is written as $$ P(w_{d,n} \mid z_{d,n}, \beta) = \beta_{w_{d,n}, z_{d,n}} $$ This formulation seems to specifically rely on knowing each word in the document.

So if I instead observe only a histogram of words $n_d$ where $n_{d,j}$ is the number of times word $j$ appears in document $d$, how would the likelihood $P(n_{d,n} \mid z_d, \beta)$ look like?

It feels like there should be a way to rewrite $P(w \mid z, \beta)$ as $P(n \mid z, \beta)$ as the word sequence is really exchangeable.

Answer 1

When you only have word counts but no documents you can generate a document from the word counts by putting the words according to their frequency in random order into the document. Then the algorithm runs smoothly without rewriting.

Answer 2

In the original LDA paper the authors note that the variational parameters $\gamma$ and $\phi$ are functions of the words $\mathbf{w}$ because the optimization of the lower bound is done for fixed $\mathbf{w}$. This means that for document $d$, topic $i$, and word positions in the document $n_j$ and $n_k$, if $w_{dn_j} = w_{dn_k}$, that is words $n_j$ and $n_k$ are the same, then:

\begin{equation} \phi_{dn_ji}(w_{dn_j}) = \phi_{dn_ki}(w_{dn_k}) \end{equation}

The M-step update for $\beta$ is

\begin{equation} \beta_{ij} \propto \sum_{d=1}^M\sum_{n=1}^{N_d} \phi_{dni}w_{dn}^j \end{equation}

Then we see that the inner sum is counting the number of times word with identity $j$ appears in document $d$ and multiplying that by some $\phi_{dji}$. So we can rewrite the M-step update as:

\begin{equation} \beta_{ij} \propto \sum_{d=1}^M \phi_{dji}n_{dj} \end{equation}

where $n_{dj}$ is the number of times the word $j$ appears in document $d$. Notice that for each document we don't have as many $\phi$ values as there are words in the document. There are only as many distinct $\phi$ values as there are distinct words.

This is why lda-c expects bag-of-words input.