I can derive an MCMC algorithm for sampling from the posterior distribution of a parameter vector of interest, but only starting with a dataset that has no missing values. The actual dataset that I want to use for inference has substantial missingness in its covariates.

One approach would be to build a more complex MCMC algorithm that, for example, first fills in the missing data with draws from the missing values' posterior predictive distribution. However, this feels intractably hard.

What I'd rather do is use an off-the-shelf method to generate multiple imputations of the dataset (such as a MICE package), then run my existing MCMC algorithm on each imputed, complete dataset and then recombine into final estimates of (for example) a posterior expectation or a posterior interval for a parameter of interest.

Is there a body of literature that attempts to solve problems in this way? Or is there a much better way to do this? Or is this approach wrong-headed or infeasible? Any pointers would be helpful.

  • $\begingroup$ why does your first proposal feel "intractably hard"? You can of course do your 2nd method (and use any kind of imputation method in the literature) but note that this type of 2 stage procedure ignores the fact that you imputed (so your estimates of standard errors will not be correct) $\endgroup$ – bdeonovic Apr 2 at 14:13
  • 1
    $\begingroup$ I have 400 covariates, with missingness in most of them. Are you aware of examples of this type where it is plausible to handle the missing data and posterior inference on the parameter simultaneously in an MCMC algorithm? $\endgroup$ – frelk Apr 5 at 0:29
  • $\begingroup$ To clarify, I am not looking for an approach that does a single imputation and then uses a single imputed dataset to perform inference on the model parameters: that would indeed ignore the fact that imputation occurred. Multiple imputation is one way of handling uncertainty due to imputation. That's why I'm wondering if there is a principled way to combined multiple imputation with procedures like MCMC so that my posterior estimates account for the imputation uncertainty. Thanks! $\endgroup$ – frelk Apr 5 at 0:34
  • $\begingroup$ Related question here, although I am looking for references that contain details: stats.stackexchange.com/questions/33596/… $\endgroup$ – frelk Apr 7 at 4:46

Are you familiar with censorship in survival analysis modeling? Censorship means that a patient was removed from the study without explicitly dying (at least in the context of survival analysis.) There are appropriate architectures to handle censorship; however, it's more a question of model architecture than the underlying MCMC sampling algorithm. For exploration of survival analysis via PyMC3 (uses HMC), see here.

In your write-up, it's not clear whether you have a survival analysis problem or something different. Nonetheless, I'd recommend approaching your problem from a model architecture perspective not an MCMC-variant-specific perspective. If you could provide some more details, we (the community) could provide some recommendations on how to handle observed and latent values (censoring.)

You didn't explicitly say that you're approaching this from a Bayesian perspective, however, MCMC is very common in the Bayesian paradigm, so I'll assume you prefer to be as Bayesian as possible. With that said, imputing values is not very Bayesian at all, as you'd be adding biases to your dataset without properly defining them via prior distributions. Censorship is the way to go so you have a "fork" so to speak that distinguishes the observed from the latent, and this will be captured nicely by your posterior distribution.


I wrote a paper arxiv.org/abs/1907.09090 that describes how the pseudo-marginal approach can impute missing data. 400 covariates sounds tough, though, to be completely honest. Depends on what kind of distributions you want to put on the columns, the number of rows, how you program everything. Intractable in your case? Probably, yes.

In section 3.3, we describe the approach (not ours) that sounds like what you want. Maybe some references in there will give you some formulas. Here’s a quote:

Multiple Imputation (MI) generates multiple complete data sets by sampling several sets of plausible values for each missing data point by sampling from the posterior predictive distribution [19], [20], [7]. The same analysis is performed separately on each data set, and the results are then combined. For example, in the context of regression analysis, the model parameters derived from each imputed dataset [sic] are combined by a simple average. The parameter variances are calculated by averaging the individual variances from each imputation, and the formula includes an additional term to capture the between-imputation variance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.