Propensity score matching after multiple imputation

Question

I refer to this paper: Hayes JR, Groner JI. "Using multiple imputation and propensity scores to test the effect of car seats and seat belt usage on injury severity from trauma registry data." J Pediatr Surg. 2008 May;43(5):924-7.

In this study, multiple imputation was performed to obtain 15 complete datasets. Propensity scores were then computed for each dataset. Then, for each observational unit, a record was chosen randomly from one of the completed 15 datasets (including the related propensity score) thereby creating a single final dataset for which was then analysed by propensity score matching.

My questions are: Is this valid way to perform propensity score matching following multiple imputation ? Are there alternative ways to do it ?

For context: In my new project, I aim to compare the effects of 2 treatment methods using propensity score matching. There is missing data and I intend to use the MICE package in R to impute missing values, then twang to do the propensity score matching, and then lme4 to analyse the matched data.

Update1:

I have found this paper which takes a different approach: Mitra, Robin and Reiter, Jerome P. (2011) Propensity score matching with missing covariates via iterated, sequential multiple imputation [Working Paper]

In this paper the authors compute propensity scores on all the imputed datasets and then pool them by averaging, which is in the spirit of multiple imputation using Rubin's rule's for a point estimate - but is it really applicable for a propensity score ?

It would be really nice if anyone on CV could provide an answer with commentary on these 2 different approaches, and/or any others....

Answer 1

The first thing to say is that, for me, method 1 (sampling) seems to be without much merit - it is discarding the benefits of multiple imputation, and reduces to single imputation for each observation, as mentioned by Stas. I can't see any advantage in using it.

There is an excellent discussion of the issues surrounding propensity score analysis with missing data in Hill (2004): Hill, J. "Reducing Bias in Treatment Effect Estimation in Observational Studies Suffering from Missing Data" ISERP Working Papers, 2004. It is downloadable from here.

The paper considers two approaches to using multiple imputation (and also other methods of dealing with missing data) and propensity scores :

• averaging of propensity scores after multiple imputation, followed by causal inference (method 2 in your post above)

• causal inference using each set of propensity scores from the multiple imputations followed by averaging of the causal estimates.

Additionally, the paper considers whether the outcome should be included as a predictor in the imputation model.

Hill asserts that while multiple imputation is preferred to other methods of dealing with missing data, in general, there is no a priori reason to prefer one of these techniques over the other. However, there may be reasons to prefer averaging the propensity scores, particularly when using certain matching algorithms. Hill did a a simulation study in the same paper and found that averaging the propensity scores prior to causal inference, when including the outcome in the imputation model produced the best results in terms of mean squared error, and averaging the scores first, but without the outcome in the imputation model, produced the best results in terms of average bias (absolute difference between estimated and true treatment effect). Generally, it is advisable to include the outcome in the imputation model (for example see here).

So it would seem that your method 2 is the way to go.

Answer 2

There might be a clash of two paradigms. Multiple imputation is a heavily model-based Bayesian solution: the concept of the proper imputation essentially states that you need to sample from the well-defined posterior distribution of the data, otherwise you are screwed. Propensity score matching, on the other hand, is a semi-parametric procedure: once you have computed your propensity score (no matter how, you could've used a kernel density estimate, not necessarily a logit model), you can do the rest by simply taking the differences between the treated and non-treated observations with the same propensity score, which is kinda non-parametric now, as there is no model left that controls for other covariates. I don't feel good about the discontinuities introduced by the literal implementation of matching (find the control with the closest possible value of the propensity score, and ignore the rest; Abadie and Imbens (2008) discussed that it makes it impossible to actually get the standard errors right in some of the matching situations). I would give more trust to the smoother approaches like weighting by the inverse propensity. My favorite reference on this is "Mostly Harmless Econometrics", subtitled "An Empiricist Companion", and aimed at economists, but I think this book should be a required reading for other social scientists, most biostatisticians, and non-bio statisticians as well so that they know how other disciplines approach data analysis.

At any rate, using only one out of 15 simulated complete data lines per observation is equivalent to a single imputation. As a result, you lose efficiency compared to all 15 completed data sets, and you can't estimate the standard errors properly. Looks like a deficient procedure to me, from any angle.

Of course, we happily sweep under the carpet the assumption that both the multiple imputation model and the propensity model are correct in the sense of having all the right variables in all the right functional forms. There is little way to check that (although I'd be happy to hear otherwise about diagnostic measures for both of these methods).

Answer 3

I can't really speak to the theoretical aspects of the question, but I'll give my experience using PS/IPTW models and multiple imputation.

1. I've never heard of someone using multiply imputed data sets and random sampling to build a single data set. That doesn't necessarily mean it's wrong but it's a strange approach to use. The data set also isn't big enough that you'd need to get creative to get around running 3-5 models instead of just one to save time and computation.
2. Rubin's rule and the pooling method is a pretty general tool. Given the pooled, multiply imputed result can be calculated using only the variance and estimates, there's no reason I can see that it couldn't be used for your project - creating the imputed data, performing the analysis on each set, and then pooling. It's what I've done, it's what I've seen done, and unless you have a specific justification not to do it, I can't really see a reason to go with something more exotic - especially if you don't understand what's going on with the method.