I'm watching a video on topic modeling by David Blei (from 27:00 of Part 2) and I don't understand how setting the $\alpha$ hyperparameter to a value close to zero basically results in a mixture model (and not mixed-membership model).

Setting $\alpha$ to a low value will result in samples of $\theta$ drawn from $Dir(\alpha)$ to be sparse, with majority of its mass concentrated in only few of its dimensions and the rest close to zero. But wouldn't the likelihood $p(D | \theta)$ dominate this effect and result in a posterior $p(\theta | D)$ where this is no longer the case?

In the video, David Blei says that, in such case, we can set $\alpha$ to be even smaller and make the prior dictate what the posterior is going to look like, but where in the formulation of the Dirichlet-multinomial (wiki) is this apparent? Without looking at too much detail, it seems that the posterior distribution, where the $\alpha$ and $counts$ are added (for each dimension), will be dictated by the $counts$ if $\alpha$ is very small.

  • $\begingroup$ "But wouldn't the likelihood p(D|θ) dominate this effect and result in a posterior p(θ|D) where this is no longer the case". I believe this is true. I know a couple of people working on new priors to induce sparsity for latent Dirichlet allocation. They told me the reason was exactly what you have described, that the likelihood washes out the effects of the prior. $\endgroup$ – AtALoss Feb 5 '17 at 6:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.