Sampling posterior of empty cluster in GMM and Gibbs

Question

Consider performing inference via a standard Gibbs sampler for a standard Gaussian Mixture Model (GMM) with $k$ components that are Gaussians $$\mathcal{N}(\mu_{k}, \sigma^{2}_{k})$$ where we assume a classical Normal-InverseGamma (NIG) prior $$\mu_{k}, \sigma_{k}^{2}\sim NIG(\mu_0, V_0, \alpha_0, \beta_0)$$ and you have observed some data $X$.

For each iteration of the Gibbs sampler, for each clustering component $k$ I would draw the corresponding parameters from the posterior $$p(\mu_{k}, \sigma^{2}_{k}, | X) = NIG(\mu_n, V_n, \alpha_n, \beta_n)$$

where the parameters $\mu_n, V_n, \alpha_n, \beta_n$ can be derived by conjugacy from $\mu_0, V_0, \alpha_0, \beta_0$ by looking at how many instances ($n$) have been assigned to cluster $k$.

Now assume that no instances have been assigned to cluster $k$ at all for a certain iteration of the Gibbs sampler. Therefore it would be correct to just sample $\mu_{k}, \sigma^{2}_{k}$ from the prior $\mu_{k}, \sigma_{k}^{2}\sim NIG(\mu_0, V_0, \alpha_0, \beta_0)$.

But what about one does not sample them and keep them "freezed" to the values obtained in the last iteration the cluster has been assigned some values? This does not sound like Gibbs sampling to me anymore, would it be still a MCMC of some sort?

Answer 1

Interesting question. As it happens, in our 1990 paper with Diebolt, we do something similar by modifying the likelihood from a regular mixture likelihood to a likelihood that is the marginal of the completed likelihood (meaning considering allocations as well as observations) such that no component can be empty or correspond to a single observation. The reason for that restriction is to allow for improper priors on all parameters of the mixture. The resulting Gibbs sampler then rejects allocations such that any component is empty or corresponds to a single observation, which means keeping the previous allocation vector $(z_1,\ldots,z_n)$ and re-updating the parameter vector, almost the dual of what you are proposing. This scheme is valid against the new likelihood. Later, in 2000, Larry Wasserman established that this model produces convergent inference about the parameters.

To answer more directly the question, I do not think that un-changing a single component parameter if no allocation occurs for that component is a valid MCMC move. All component parameters have to remain the same.