Dirichlet Process Hyperparameter Estimation with Sampling

Question

I have a dirichlet process for which I need to learn the concentration (strength) hyperparameter (with gamma prior on it). One way of doing is via maximizing the Likelihood. Another way of doing this is via Sampling. I wonder how this can be done via sampling (i.e. Slice, Gibbs, ...)? Any reference to papers discussing this is more than enough for me.

Answer 1

Escobar and West give the standard Gibbs update for a gamma prior on the concentration parameter $\alpha$. Also note that the likelihood of the $\alpha$ given a partition $\mathscr P = \{b_1, b_2, \ldots, b_P\}$ of the data is given by $$ \frac{\Gamma(\alpha) \alpha^P}{\Gamma(\alpha + N)} \prod_{p = 1} ^ P \Gamma(|b_p|), $$ so you can use this to construct a Metropolis-Hastings update for a non-conjugate prior if you desire (of course, make sure that you work on in log-probability space).

It is also possible to estimate $\alpha$ by maximum marginal likelihood, although this is a little trickier. It is possible for the MLE of $\alpha$, to be $\hat \alpha = \infty$ or $\hat \alpha = 0$. It also happens to be the case that the Fisher information about $\alpha$ accrues at roughly a logrithmic rate, so you shouldn't expect the MLE to behave like other MLE's you've seen (the log-likelihood is often nowhere near quadratic even in moderate/large sample sizes). It is also possible for the likleihood of $\alpha$ to be multi-modal, as shown by Kyung et al, although I've never seen this in practice.

If you really want to use maximum likelihood, the MLE is defined by the moment matching condition $E_\alpha[P] = E_\alpha [P \mid \mbox{Data}]$ where $P$ is the number of groups. To get the MLE, one can iteratively update estimates $\hat \alpha^{(t)}$ to satisfy this moment matching condition, running a seperate chain for each $t$, setting $$ E_{\hat \alpha^{(t+1)}} [P] = E_{\hat \alpha^{(t)}} [P \mid \mbox{Data}]. $$ $E_{\alpha}[P]$ is given in closed form by $\sum_{i = 1} ^ N \frac{\alpha}{\alpha + i - 1}$, and $E_{\hat \alpha^{(t)}}[P \mid \mbox{Data}]$ is available from the output of the Markov chain.