I am working on the so called label switching problem in Bayesian inference with Gaussian Mixture Models.

To put in a nutshell, when your favourite MCMC samplers estimates the parameters of your GMM, it does it without labelling the modes. That is to say that the parameters estimates are similar always up to a permutation, which prevents from taking the mean of parameters estimates as a good final estimator.

A solution is to relabel each sample of parameters. A way to do so is to use the pivot algorithm describe here https://arxiv.org/pdf/1503.02271.pdf. You define a pivot as the most likely sample of parameters and then align each sample of parameters to this pivot by maximizing the dot product between the pivot and the sample.

To make it clear : let's say I have a GMM with 2 components in $\mathbb{R}$. So my MCMC returns me samples of the form $\chi=(\mu_1,\mu_2, \sigma_1, \sigma_2)$. After defining your pivot $\chi^*$ as the likeliest sample of parameters, for each sample $\chi$ you want to find a permutation $\tau$ that maximizes $<\chi_\tau, \chi^*>$.

I would like to know when this method fails. Do you have any idea of GMM or distributions when this method is unable to recover the permutation and then failing to have good final estimators for the parameters ?

  • $\begingroup$ You need to define what you mean by "the" permutation, i.e., what is the criterion behind choosing one permutation vs. the other. $\endgroup$ – Xi'an May 12 at 14:16
  • $\begingroup$ What I want to find is a case where this relabelling methods fails to estimate properly the parameters of a GMM. So maybe a case when this methods fails to recover the right permutation associated to each sample (for instance for a sample $\chi$ this permutation is defined as $argmin <\chi_\tau,\chi^*>$, and this permutation is different for every sample) and therefore fails to recover the parameters of the GMM $\endgroup$ – MarcM May 13 at 3:09
  • $\begingroup$ This does not answer your question, but why not (1) define the mean prior as ordered vector, or (2) give a bounded prior to both mu parameter, such that the upper limit of mu1 is lower or equal than the lower limit of mu2? In both cases you can transform the parameter to an unbounded space for sampling procedure. $\endgroup$ – MrVengeanZe May 13 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.