5
$\begingroup$

I'm using a Gibbs sampler for Latent Dirichlet allocation as described by Griffiths and Steyvers (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC387300/). The sampling of a new topic $j$ for word $i$ is described by

$P(z_i = j | \mathbf{z}_{-i}, \mathbf{w}) \propto \frac{n^{(w_i)}_{-i,j} + \beta}{n^{(\bullet)}_{-i,j} + W \beta} \frac{n^{(d_i)}_{-i,j} + \alpha}{n^{(d_i)}_{-i} + T \alpha}$

For the sake of the question, let's focus on $\frac{n^{(d_i)}_{-i,j} + \alpha}{n^{(d_i)}_{-i} + T \alpha}$, which describes how a word's topic is sampled in relation to the current topic counts. The term $n^{(d_i)}_{-i,j}$ is the number of words in the document (not including the word we're updating) assigned to topic $j$, while $n^{(d_i)}_{-i}$ is the number of total words in the document not including the current word, $T$ is the number of topics, and $\alpha$ is the concentration parameter for the Dirichlet prior.

In many applications, it's reasonable for documents to only contain one or two topics, which leads to a very small concentration parameter $\alpha$, ($<0.01$). It seems that when $\alpha$ gets very small, the prior doesn't "exert much influence" in the sampling. The topics are mostly sampled in proportion to existing counts with only a very small $\alpha$ term added in.

In my experience, this doesn't tend to steer the sampler toward distributions that favor only one or two topics (or at least not quickly at all), as the sampler just samples in proportion to the initial topic distribution. In the extreme case of $\alpha =0$, there should be exactly one topic per document, but even after thousands and thousands of iterations, my sampler does not seem to approach such a distribution.

Is there any way for a small Dirichlet concentration parameter to "exert more influence" in Gibbs sampling?

Improve this question
$\endgroup$
5
  • $\begingroup$ What happens if you start the mcmc chain with 1 topic per document? Does it ever leave that topic initially assigned? $\endgroup$
    – probabilityislogic
    Commented Dec 11, 2013 at 21:14
  • $\begingroup$ If the $\alpha$ is zero, then no, it sticks with the same initial topic because that's the only topic with a non-zero probability. With a small $\alpha$, it will stray a little, but not far, even when there really should be a non-negligible second topic. $\endgroup$
    – Ben
    Commented Dec 11, 2013 at 22:57
  • $\begingroup$ I think any markov chain with $\alpha <1 $ will struggle as there are multiple modes that need to be visited and this gibbs sampler just isn't able to find them by making "local" moves. Better to control the number of topics with T directly $\endgroup$
    – probabilityislogic
    Commented Dec 12, 2013 at 7:25
  • $\begingroup$ Since the dirichlet parameters describe a prior distribution, isn't this just an indication that the data presents strong evidence against a sparse document-topic prior? I have found that on large datasets, the dirichlet parameters don't have a dramatic effect on the model (although they do make a difference, and are worth tuning for reasons stated here). MALLET's topic model gives you the option to estimate priors from the data, and the priors converge to a stable point pretty quickly. $\endgroup$
    – senderle
    Commented Nov 18, 2014 at 18:11
  • $\begingroup$ Note also that the dirichlet distribution is only defined for a > 0, so passing a = 0 won't give sensible results. $\endgroup$
    – senderle
    Commented Nov 18, 2014 at 18:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.