1
$\begingroup$

I've always implemented DP MM's using chinese restaurant process, which necessitates sequential sampling of cluster assignments (as cluster weights depend on current number of observations in cluster, specifically not including observation $i$. That is, $$ P(\gamma_i = j\mid\alpha,\gamma_{\neg i},\theta) \propto \begin{cases} n_j^{\neg i}L(y_i\mid\theta_j) &\text{ for }j = 1,\ldots,J^{\neg i}\\ \alpha\int_{\theta}L(y_i\mid\theta)\text{d}G_0(\theta) & \text{ for }j = J^{\neg i} + 1 \end{cases} $$ This means cluster weights depend on current counts. In testing, this sampling of cluster assignments represents the bulk of the CPU time of the sampler.

I'm working to implement a Pitman-Yor process, which in implementation isn't particularly different from the Dirichlet process. However, while reading literature, I see in the blocked Gibbs sampler using a stick-breaking representation (looking at pg. 552 of Bayesian Data Analysis, 3rd edition), it seems to allow simultaneous sampling of cluster assignments.

$$ \begin{aligned} \lambda_j\mid\alpha,\gamma &\sim \text{Beta}\left(1 + n_j,\alpha + \sum_{k = j + 1}^N n_k\right)\\ \pi_j &= \lambda_j\prod_{k=1}^{j-1}(1-\lambda_k)\\ P(\gamma_i = j\mid\pi,\theta) &\propto \pi_j L(y_i\mid\theta_j) \end{aligned} $$

Note that $\pi_j$ does not explicitly depend on observation $i$. Indeed, in BDA, first the cluster assignments are sampled, then the weights are sampled, and finally the cluster parameters are sampled. If this is valid, this would vastly increase sampling speed for my analysis.

Is my understanding correct? Is the simultaneous sampling of cluster assignments valid? What am I giving up by implementing such a method?

$\endgroup$

1 Answer 1

1
$\begingroup$

The stick-breaking representation means that you don't need to keep track of the cluster sizes $n_j$ when updating the cluster assignments. So, conditioned on the stick lengths and cluster parameters, the cluster assignments should be independent, and can be updated in block.

I suppose you have to represent stick-lengths and cluster parameters here, which with a CRP (and a conjugate model) are not necessary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.