Fitting/Inference for Dirichlet Process/CRP for clustering

Question

Please excuse my ignorance on this topic, I don't have much experience in nonparametric Bayes. I read about Dirichlet process clustering and the Chinese Restaurant Process analogies. I think I understand the model, and how one can generate clustered data points from it.

But how does one fit this given data for inference? Its greatest utility seems to be inferring an ideal cluster count without prespecification, but every source I've read about the topic seems to be more about the model rather than how it is applied and doesn't really give examples with data.

Answer 1

The most intuitive way for inference is based on the CRP generative process. Given some data $X$ of length $N$ and labels $Z$, you would like to know if some sample belong to table $t_k$, in this analogy, each table is associated with a mixture component $\theta_k$, we denote $|t_k^{-1}|$ as the number of customers siting at table $k$, minus the $i^{th}$ customer if he is sitting there: $$ \Pr(z_i = k|x,Z^{-i},(\theta_k)_1^\infty )\propto \frac{|t_k^{-1}|}{N-1+\alpha}\cdot f_\theta(x_i|\theta_k) $$ In words: The probability of sample $x_i$ sitting at table $t_k$ (or belonging to mixture component $\theta_k$) is proportional to the number of customers siting at the table, and the likelihood of $x_i$ beloning to componenet $\theta_k$ (which is the likelihood function, and depends on the component type). In addition, there is always the chance of opening a new table: $$ \Pr(z_i = k_{new}|x,Z^{-i},(\theta_k)_1^\infty )\propto \frac{\alpha}{N-1+\alpha}\cdot f_\theta(x_i|\theta_{new}) $$ Where in this case $\theta_{new}$ is a sample from your base measure $H$.

You update the component parameters according to the samples belonging to each component. This process is very slow, as each label depends on the rest of the entire model, but in my opinion, it is intuitive.

There are other methods for inference, either by sampling or by variational inference, which can be very fast.

Answer 2

(I think Dinari's answer is correct, but I want to try to target the difficulty faced by OP a bit more.)

every source I've read about the topic seems to be more about the model rather than how it is applied and doesn't really give examples with data.

The Dirichlet process (DP) or CRP is only the prior in a Dirichlet process mixture model. It is not a clustering algorithm as you seem to think. Therefore, if you search for DP or CRP, you might only get details about the process itself, but not on how to use it in mixture models.

More precisely, in a mixture model, the cluster assignment variables are latent variables and each cluster is characterised by a set of parameter (for example, mean and variance if your base distribution is a Gaussian). The DP is used as a prior over cluster assignments. It is easy to use with Gibbs sampling because the posterior of a single assignment variable given the other assignments is very easy to compute. (See Dinari's formulas.)

If you are using a Gaussian, you will have a mean and variance parameter for each cluster (and in our case, the number of clusters is sampled and can vary, thanks to the CRP). You would take the mean parameter for the cluster $i$ to be simply the mean of the datapoints which are assigned to this cluster $i$.

I am not sure about the following, please correct me if I'm wrong / approve if I'm right: You can fit finite mixture models with EM. In the E-step, you compute the expectation of each cluster assignment and from there find the parameters that maximize it. Here, you cannot computing the expectation under the posterior, so instead, you sample from it and maximize. So maybe it is a variant of EM with sampling at the E-step.