# Running Latent Dirichlet Allocation (LDA) on word counts

I have difficulties understanding the VB implementation lda-c. In particular, the method expects as input a bag-of-words representation of documents, where distinct words appearing in a document are mapped to the number of their occurence in that document. The generative model, however, seems to be specified in terms of a sequence of words per document denoted by $w_{d, n} \in \{1,\ldots,V\}$ for the $n$-th word taken from our vocabulary of size $V$ in the $d$-th document. Each word is assigned a latent topic variable $z_{d,n} \in \{1, \ldots, K\}$. Given $z_{d,n}$ and word distributions $\beta_k$ per topic $k$, the likelihood is written as $$P(w_{d,n} \mid z_{d,n}, \beta) = \beta_{w_{d,n}, z_{d,n}}$$ This formulation seems to specifically rely on knowing each word in the document.

So if I instead observe only a histogram of words $n_d$ where $n_{d,j}$ is the number of times word $j$ appears in document $d$, how would the likelihood $P(n_{d,n} \mid z_d, \beta)$ look like?

It feels like there should be a way to rewrite $P(w \mid z, \beta)$ as $P(n \mid z, \beta)$ as the word sequence is really exchangeable.

• Look at section 3.1 of the original paper. Topics are exchangeable but words aren't necessarily May 27, 2015 at 12:32

When you only have word counts but no documents you can generate a document from the word counts by putting the words according to their frequency in random order into the document. Then the algorithm runs smoothly without rewriting.

In the original LDA paper the authors note that the variational parameters $\gamma$ and $\phi$ are functions of the words $\mathbf{w}$ because the optimization of the lower bound is done for fixed $\mathbf{w}$. This means that for document $d$, topic $i$, and word positions in the document $n_j$ and $n_k$, if $w_{dn_j} = w_{dn_k}$, that is words $n_j$ and $n_k$ are the same, then:

$$\phi_{dn_ji}(w_{dn_j}) = \phi_{dn_ki}(w_{dn_k})$$

The M-step update for $\beta$ is

$$\beta_{ij} \propto \sum_{d=1}^M\sum_{n=1}^{N_d} \phi_{dni}w_{dn}^j$$

Then we see that the inner sum is counting the number of times word with identity $j$ appears in document $d$ and multiplying that by some $\phi_{dji}$. So we can rewrite the M-step update as:

$$\beta_{ij} \propto \sum_{d=1}^M \phi_{dji}n_{dj}$$

where $n_{dj}$ is the number of times the word $j$ appears in document $d$. Notice that for each document we don't have as many $\phi$ values as there are words in the document. There are only as many distinct $\phi$ values as there are distinct words.

This is why lda-c expects bag-of-words input.