Best way to combine MCMC inference with multiple imputation?

Question

I can derive an MCMC algorithm for sampling from the posterior distribution of a parameter vector of interest, but only starting with a dataset that has no missing values. The actual dataset that I want to use for inference has substantial missingness in its covariates.

One approach would be to build a more complex MCMC algorithm that, for example, first fills in the missing data with draws from the missing values' posterior predictive distribution. However, this feels intractably hard.

What I'd rather do is use an off-the-shelf method to generate multiple imputations of the dataset (such as a MICE package), then run my existing MCMC algorithm on each imputed, complete dataset and then recombine into final estimates of (for example) a posterior expectation or a posterior interval for a parameter of interest.

Is there a body of literature that attempts to solve problems in this way? Or is there a much better way to do this? Or is this approach wrong-headed or infeasible? Any pointers would be helpful.

Answer 1

Are you familiar with censorship in survival analysis modeling? Censorship means that a patient was removed from the study without explicitly dying (at least in the context of survival analysis.) There are appropriate architectures to handle censorship; however, it's more a question of model architecture than the underlying MCMC sampling algorithm. For exploration of survival analysis via PyMC3 (uses HMC), see here.

In your write-up, it's not clear whether you have a survival analysis problem or something different. Nonetheless, I'd recommend approaching your problem from a model architecture perspective not an MCMC-variant-specific perspective. If you could provide some more details, we (the community) could provide some recommendations on how to handle observed and latent values (censoring.)

You didn't explicitly say that you're approaching this from a Bayesian perspective, however, MCMC is very common in the Bayesian paradigm, so I'll assume you prefer to be as Bayesian as possible. With that said, imputing values is not very Bayesian at all, as you'd be adding biases to your dataset without properly defining them via prior distributions. Censorship is the way to go so you have a "fork" so to speak that distinguishes the observed from the latent, and this will be captured nicely by your posterior distribution.

Answer 2

I wrote a paper arxiv.org/abs/1907.09090 that describes how the pseudo-marginal approach can impute missing data. 400 covariates sounds tough, though, to be completely honest. Depends on what kind of distributions you want to put on the columns, the number of rows, how you program everything. Intractable in your case? Probably, yes.

In section 3.3, we describe the approach (not ours) that sounds like what you want. Maybe some references in there will give you some formulas. Here’s a quote:

Multiple Imputation (MI) generates multiple complete data sets by sampling several sets of plausible values for each missing data point by sampling from the posterior predictive distribution [19], [20], [7]. The same analysis is performed separately on each data set, and the results are then combined. For example, in the context of regression analysis, the model parameters derived from each imputed dataset [sic] are combined by a simple average. The parameter variances are calculated by averaging the individual variances from each imputation, and the formula includes an additional term to capture the between-imputation variance.