Dirichlet Process Hyperparameter Estimation with Sampling I have a dirichlet process for which I need to learn the concentration (strength) hyperparameter (with gamma prior on it). One way of doing is via maximizing the Likelihood. Another way of doing this is via Sampling. I wonder how this can be done via sampling (i.e. Slice, Gibbs, ...)? Any reference to papers discussing this is more than enough for me. 
 A: Escobar and West give the standard Gibbs update for a gamma prior on the concentration parameter $\alpha$. Also note that the likelihood of the $\alpha$ given a partition $\mathscr P = \{b_1, b_2, \ldots, b_P\}$ of the data is given by 
$$
\frac{\Gamma(\alpha) \alpha^P}{\Gamma(\alpha + N)} \prod_{p = 1} ^ P \Gamma(|b_p|), 
$$
so you can use this to construct a Metropolis-Hastings update for a non-conjugate prior if you desire (of course, make sure that you work on in log-probability space). 
It is also possible to estimate $\alpha$ by maximum marginal likelihood, although this is a little trickier. It is possible for the MLE of $\alpha$, to be $\hat \alpha = \infty$ or $\hat \alpha = 0$. It also happens to be the case that the Fisher information about $\alpha$ accrues at roughly a logrithmic rate, so you shouldn't expect the MLE to behave like other MLE's you've seen (the log-likelihood is often nowhere near quadratic even in moderate/large sample sizes). It is also possible for the likleihood of $\alpha$ to be multi-modal, as shown by Kyung et al, although I've never seen this in practice. 
If you really want to use maximum likelihood, the MLE is defined by the moment matching condition $E_\alpha[P] = E_\alpha [P \mid \mbox{Data}]$ where $P$ is the number of groups. To get the MLE, one can iteratively update estimates $\hat \alpha^{(t)}$ to satisfy this moment matching condition, running a seperate chain for each $t$, setting 
$$
E_{\hat \alpha^{(t+1)}} [P] = E_{\hat \alpha^{(t)}} [P \mid \mbox{Data}].
$$ $E_{\alpha}[P]$ is given in closed form by $\sum_{i = 1} ^ N \frac{\alpha}{\alpha + i - 1}$, and $E_{\hat \alpha^{(t)}}[P \mid \mbox{Data}]$ is available from the output of the Markov chain. 
