3
$\begingroup$

I am running the c code for LDA provided on David Blei's website. The code outputs several files. The output file final.gamma is supposed to include the "Variational Posterior Dirichlets". If I understand this correctly (which it is very likely I do not) it is supposed to be for every document the probability that that document was generated by each of the k topics. If my understanding is correct then the sum of the posteriors for each document should add up to 1. i.e. the sum over the k topics of the probability that that topic generated the document should be 1 (every article is assumed to be generated by some mixture of the topics). I ran LDA with 2 topics and 20 documents and got the following final.gamma file:

0.0163467211 439.0163467211
476.0163467211 0.0163467211
0.0163467211 141.0163467211
151.0163467211 0.0163467211
0.0163467211 526.0163467211
0.0163467211 78.0163467211
52.3257109467 314.7069824954
0.0163467211 356.0163467211
2448.0163467211 0.0163467211
0.0163467211 356.0163467211
0.0163467211 554.0163467211
0.0163467211 632.0163467211
373.0163467211 0.0163467211
0.0163467211 533.0163467211
726.0163467211 0.0163467211
0.0163467211 501.0163467211
38.8105963538 347.2220970883
470.0163467211 0.0163467211
0.0163467211 437.0163467211
723.0163467211 0.0163467211

As you can see for each document there are two numbers representing the two topics that generated that article. These numbers do not sum up to 1, what do they mean?

For this output I ran LDA on twenty documents the first ten relating to baseball and the second ten relating to football. I would expect LDA to assign the first ten documents similar posteriors and the second ten documents similar posteriors but that is not the case. The words that the topics generated are clearly split on the football/baseball axis i.e. topic 0 generates only baseball words and topic 1 only football words so I would expect the posteriors to work.

Is this a bug in the LDA implementation or am I misunderstanding the meaning of the posteriors?

$\endgroup$
2
$\begingroup$

No, this is not a bug.

To convert the $\gamma$ matrix in final.gamma file into probabilities, you have to

  1. subtract your final $\alpha$ off from $\gamma$
  2. normalize each row of $\gamma$, i.e., $\gamma_i$, by $$\gamma_i=\frac{\gamma_{i}}{\Sigma_j\gamma_{ij}}$$

For more details, see this thread in their mailing list.

Update

As per the recent discussions in the mailing list, David Blei (the original author) says it is optional to subtract the $\alpha$ off. They (sometimes) do so to make the topic distributions of the documents less smooth.

$\endgroup$
1
  • $\begingroup$ @BenjyKessler haha, almost one year ago. I am sorry that I didn't come answer it earlier. $\endgroup$ – Sibbs Gambling Nov 19 '14 at 3:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.