Consider performing inference via a standard Gibbs sampler for a standard Gaussian Mixture Model (GMM) with $k$ components that are Gaussians $$\mathcal{N}(\mu_{k}, \sigma^{2}_{k})$$ where we assume a classical Normal-InverseGamma (NIG) prior $$\mu_{k}, \sigma_{k}^{2}\sim NIG(\mu_0, V_0, \alpha_0, \beta_0)$$ and you have observed some data $X$.

For each iteration of the Gibbs sampler, for each clustering component $k$ I would draw the corresponding parameters from the posterior $$p(\mu_{k}, \sigma^{2}_{k}, | X) = NIG(\mu_n, V_n, \alpha_n, \beta_n)$$

where the parameters $\mu_n, V_n, \alpha_n, \beta_n$ can be derived by conjugacy from $\mu_0, V_0, \alpha_0, \beta_0$ by looking at how many instances ($n$) have been assigned to cluster $k$.

Now assume that no instances have been assigned to cluster $k$ at all for a certain iteration of the Gibbs sampler. Therefore it would be correct to just sample $\mu_{k}, \sigma^{2}_{k}$ from the prior $\mu_{k}, \sigma_{k}^{2}\sim NIG(\mu_0, V_0, \alpha_0, \beta_0)$.

But what about one does not sample them and keep them "freezed" to the values obtained in the last iteration the cluster has been assigned some values? This does not sound like Gibbs sampling to me anymore, would it be still a MCMC of some sort?

  • $\begingroup$ Irrespective of whether it is valid (it is not), it is a bad idea to freeze. Imagine you initialize in a way that makes one cluster have really bad parameters; because of this, the cluster is empty. Now you are stuck with these bad initial values effectively forever because you are not changing them. If you update them using the prior, you should eventually by chance have them be reasonable values, at which point you might get a new cluster appearing. $\endgroup$ – guy Jul 16 '18 at 20:35
  • $\begingroup$ actually I was saying to freezing them to the last configuration where the cluster had been assigned some instances $\endgroup$ – rano Jul 17 '18 at 20:41
  • $\begingroup$ The point was to give an example to show why it is much better to update them. The mixing of the chain (assuming it actually corresponds to some stationary distribution) will potentially be made much worse by freezing; I was just giving an extreme example. I cannot see any upside in freezing, aside from saving a pretty negligible amount of compute time. $\endgroup$ – guy Jul 17 '18 at 23:03
  • $\begingroup$ I see what you mean, but the extreme example is somehow outside the contemplated cases : ) Freezing the allocations might be interesting on the other hand, see @Xi'an answer below $\endgroup$ – rano Jul 19 '18 at 4:29

Interesting question. As it happens, in our 1990 paper with Diebolt, we do something similar by modifying the likelihood from a regular mixture likelihood to a likelihood that is the marginal of the completed likelihood (meaning considering allocations as well as observations) such that no component can be empty or correspond to a single observation. The reason for that restriction is to allow for improper priors on all parameters of the mixture. The resulting Gibbs sampler then rejects allocations such that any component is empty or corresponds to a single observation, which means keeping the previous allocation vector $(z_1,\ldots,z_n)$ and re-updating the parameter vector, almost the dual of what you are proposing. This scheme is valid against the new likelihood. Later, in 2000, Larry Wasserman established that this model produces convergent inference about the parameters.

To answer more directly the question, I do not think that un-changing a single component parameter if no allocation occurs for that component is a valid MCMC move. All component parameters have to remain the same.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.