2
$\begingroup$

I have a dirichlet process for which I need to learn the concentration (strength) hyperparameter (with gamma prior on it). One way of doing is via maximizing the Likelihood. Another way of doing this is via Sampling. I wonder how this can be done via sampling (i.e. Slice, Gibbs, ...)? Any reference to papers discussing this is more than enough for me.

$\endgroup$
  • $\begingroup$ For sampling (eg Gibbs) I believe you need a prior. Do you have one? Are you just asking how Gibbs sampling works in this case or something else? $\endgroup$ – gung - Reinstate Monica Sep 30 '14 at 0:37
  • 1
    $\begingroup$ Well assume that the hyperparameter has gamma prior on it. I wonder how Gibbs sampling can be used to estimate the hyperparameter of DP. $\endgroup$ – user3639557 Sep 30 '14 at 0:42
  • 1
    $\begingroup$ The question can be rephrased to a more general one: "how to use samples to estimate the hyperparameters". $\endgroup$ – user3639557 Sep 30 '14 at 0:48
2
$\begingroup$

Escobar and West give the standard Gibbs update for a gamma prior on the concentration parameter $\alpha$. Also note that the likelihood of the $\alpha$ given a partition $\mathscr P = \{b_1, b_2, \ldots, b_P\}$ of the data is given by $$ \frac{\Gamma(\alpha) \alpha^P}{\Gamma(\alpha + N)} \prod_{p = 1} ^ P \Gamma(|b_p|), $$ so you can use this to construct a Metropolis-Hastings update for a non-conjugate prior if you desire (of course, make sure that you work on in log-probability space).

It is also possible to estimate $\alpha$ by maximum marginal likelihood, although this is a little trickier. It is possible for the MLE of $\alpha$, to be $\hat \alpha = \infty$ or $\hat \alpha = 0$. It also happens to be the case that the Fisher information about $\alpha$ accrues at roughly a logrithmic rate, so you shouldn't expect the MLE to behave like other MLE's you've seen (the log-likelihood is often nowhere near quadratic even in moderate/large sample sizes). It is also possible for the likleihood of $\alpha$ to be multi-modal, as shown by Kyung et al, although I've never seen this in practice.

If you really want to use maximum likelihood, the MLE is defined by the moment matching condition $E_\alpha[P] = E_\alpha [P \mid \mbox{Data}]$ where $P$ is the number of groups. To get the MLE, one can iteratively update estimates $\hat \alpha^{(t)}$ to satisfy this moment matching condition, running a seperate chain for each $t$, setting $$ E_{\hat \alpha^{(t+1)}} [P] = E_{\hat \alpha^{(t)}} [P \mid \mbox{Data}]. $$ $E_{\alpha}[P]$ is given in closed form by $\sum_{i = 1} ^ N \frac{\alpha}{\alpha + i - 1}$, and $E_{\hat \alpha^{(t)}}[P \mid \mbox{Data}]$ is available from the output of the Markov chain.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Great! Also, I noticed that Slice sampling does an amazing job for drawing samples. It doesn't have the heavy MH rejection criteria, and instead uses cheap corrections. Plus, that if you don't know how to draw samples from conditionals, you can still use slice sampling and draw samples which will otherwise will be impossible via Gibbs. $\endgroup$ – user3639557 Sep 30 '14 at 3:36
  • $\begingroup$ @user3639557 Slice sampling is a special case of MH, with an acceptance probability of 1. $\endgroup$ – guy Sep 30 '14 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.