Bayesian MCMC and Only Updating Some Variables at a Time

Question

I want to do Bayesian MCMC on a Gaussian Mixture Model. But, I want to update the means, weights, and covariance matrix for a single component separate from the others. Would there be the issue of variables not updating often enough? Like, if I have four components, then each variable can only be updated every four iterations at most. But then again, I can't see how this would compromise detailed balance. Is there literature on this? Thanks!

Answer 1

Sampling only one variable (or more generally, a subset of variables), conditioned on the most recent samples of all the others, is what is done in Gibbs sampling, which is a very well established MCMC method. Gibbs sampling is typically applied when the conditional distribution of each variable (or most variables) is easy to sample from, e.g. because it has a simple form that enables direct sampling (like a Gaussian). In these cases, Gibbs can be very efficient.

However, it is not necessary that we be able to sample from the conditionals directly. You can use any sampling method. For instance, if you use Metropolis-Hastings to sample from each conditional in turn, you get what is sometimes called "Metropolis-within-Gibbs". You also need not sample only a single variable; you can sample any subset of variables given all the others, as long as you regularly update all variables.

In any case, you can rest assured that what you want to do is perfectly valid. Variables "not updating often enough" is not a problem as far as detailed balance is concerned. Your chain will approximate the target distribution. Whether this method is the most efficient for your particular problem is another question. Sometimes it is faster to sample all variables at once, sometimes it isn't - it all depends on the specifics of your case.

What you do probably want to do (in case you hadn't realized this yet) is only store every completely new sample of all variables. If you store the intermediate samples, where only some of the variables have been updated, you get very strong redundancy between your samples. This isn't fundamentally a problem - autocorrelation in MCMC never is (as long as you obtain enough samples). It's just a bit of a waste of memory and/or storage to retain samples with very strong autocorrelation, and will also slow down any calculations you want to do across your samples (e.g. computing summary statistics, marginals, etc.).