0
$\begingroup$

All the code and data for this question is on GitHub (stackexchange.R script). I've got multivariate Bernoulli data that I'd like to analyse using Bayesian Mixture Models. I've simulated 1,000 observations of 5 binary variables, split into 2 clusters in the ratio 0.7:0.3, each having different Bernoulli parameter vectors $\theta^5$.

I first implemented a Collapsed Gibbs Sampler for a finite mixture model with $K=2$, following eq 21 in van der Maaten, this code is in src/collapsed_gibbs.cpp. Sampling from this model has correctly identified the 2 clusters with their different $\theta$ vectors and their mixing proportions, as shown below.

enter image description here

Now I want to extend this to use a Dirichlet Process prior on $K$, with the sampling probabilities in Eqs 23 and 25 in van der Maaten using the Chinese Restaurant Process formulation (CRP). There isn't much difference in the sampling probabilities, except that that the prior on the known clusters changes from $\frac{N_{-nk} + \frac{\alpha}{K}}{N-1+\alpha}$ to $\frac{N_{-nk}}{N-1 + \alpha}$ and the prior on a new cluster is $\frac{\alpha}{N-a+\alpha}$. The likelihood for known clusters doesn't change, while for new clusters for an observation $i$ on a variable $d$ it is $\frac{\beta}{\beta + \gamma}$ when $x_{nd} = 1$ and $\frac{\gamma}{\beta + \gamma}$ when $x_{id} = 0$, where $\beta$ and $\gamma$ are the Beta priors on $\theta$. I'm using symmetric priors $\beta = \gamma = 0.5$, so the log-likelihood for an observation $i$ over all datapoints collapses to $5 \log(\beta) = 5 \log(\gamma) = 5\log(0.5)$

Sampling from this model produces something vaguely representative of the data, in that the 2 clusters are identified, but their label changes every few thousand samples, i.e. how the top trace below changes from the red component to a pink component around sample 6000. There's also quite a bit of noise in new clusters being generated with very few members.

Is there anything immediately obvious that I'm doing wrong with my sampling? I can't understand why after 6000 samples a new cluster is formed that sucks away all the members from an existing one that had 70% of the observations in it, to me it seems like the assignment probabilities are wrong but I can't see where this has occurred.

I'lll also say that I've put a $\text{Gamma}(1,1)$ prior on the concentration parameter $\alpha$, but I was seeing this behaviour when it was fixed at $\alpha=1$.

enter image description here

$\endgroup$
  • 1
    $\begingroup$ I didn't check your math but the cluster labels are not identified so this behavior seems expected. They aren't identified in the finite mixture either... You might want to try running the finite mixture with K=10 or something and I'd guess you'd see something similar. With K=2, and N=1000 it is really unlikely for the labels to switch. $\endgroup$ – HStamper Jul 29 at 16:49
  • 1
    $\begingroup$ Yes, but I don't know if it's dreaded. Since the labels are identified this is what should happen if your MCMC is mixing properly. It doesn't cause a problem for "person-level" inference for instance. $\endgroup$ – HStamper Jul 30 at 21:54
  • 1
    $\begingroup$ It appears so, although I haven't ever been serious about trying to address it. When your model admits an possibly infinite number of undetermined clusters it is hard to make labels stick. It seems to me that one could defend doing something fairly simple (albeit a little adhoc) by tying the labels to the cluster membership of particular "persons". $\endgroup$ – HStamper Jul 31 at 1:08
  • 1
    $\begingroup$ Although not recommended by the authors you cited, it seems like you might get satisfactory results just relabeling draws using the sort of "percent in cluster". $\endgroup$ – HStamper Jul 31 at 1:12
  • 1
    $\begingroup$ If you aren't aware, there is another way to fit (almost) the same DP model called "truncated DP mixture" (Ishwaran and James 2000?) It doesn't marginalize out the $\pi_k$ and the $\pi_k$ are stochastically decreasing, so that may give cleaner results for this problem. $\endgroup$ – HStamper Jul 31 at 1:17
1
$\begingroup$

This "label-switching" behavior is expected. We would like the labels to stick to the cluster-specific Bernoulli parameters, but since the clusters are nothing more than possible partitions of individuals, cluster labels are not identified. The fact that you don't see the switching in the first case is because the chain is stuck in a particular configuration; it could just as easily ended up swapping labels/colors given different initial values.

To sum up, label-switching is a feature of finite mixture models as well as infinite mixture models, such as Dirichlet Process mixture models. As you observed, increasing K to larger values will produce the same kind of behavior. There is a fair amount of literature around ways to "fix" it. Probably the best way would be to relabel the draws post-processing. Note also that marginal inference for individuals as well as for the mixing distribution is not impacted by label-switching.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.