# Propensity score matching after multiple imputation

I refer to this paper: Hayes JR, Groner JI. "Using multiple imputation and propensity scores to test the effect of car seats and seat belt usage on injury severity from trauma registry data." J Pediatr Surg. 2008 May;43(5):924-7.

In this study, multiple imputation was performed to obtain 15 complete datasets. Propensity scores were then computed for each dataset. Then, for each observational unit, a record was chosen randomly from one of the completed 15 datasets (including the related propensity score) thereby creating a single final dataset for which was then analysed by propensity score matching.

My questions are: Is this valid way to perform propensity score matching following multiple imputation ? Are there alternative ways to do it ?

For context: In my new project, I aim to compare the effects of 2 treatment methods using propensity score matching. There is missing data and I intend to use the MICE package in R to impute missing values, then twang to do the propensity score matching, and then lme4 to analyse the matched data.

Update1:

I have found this paper which takes a different approach: Mitra, Robin and Reiter, Jerome P. (2011) Propensity score matching with missing covariates via iterated, sequential multiple imputation [Working Paper]

In this paper the authors compute propensity scores on all the imputed datasets and then pool them by averaging, which is in the spirit of multiple imputation using Rubin's rule's for a point estimate - but is it really applicable for a propensity score ?

It would be really nice if anyone on CV could provide an answer with commentary on these 2 different approaches, and/or any others....

The first thing to say is that, for me, method 1 (sampling) seems to be without much merit - it is discarding the benefits of multiple imputation, and reduces to single imputation for each observation, as mentioned by Stas. I can't see any advantage in using it.

There is an excellent discussion of the issues surrounding propensity score analysis with missing data in Hill (2004): Hill, J. "Reducing Bias in Treatment Effect Estimation in Observational Studies Suffering from Missing Data" ISERP Working Papers, 2004. It is downloadable from here.

The paper considers two approaches to using multiple imputation (and also other methods of dealing with missing data) and propensity scores :

• averaging of propensity scores after multiple imputation, followed by causal inference (method 2 in your post above)

• causal inference using each set of propensity scores from the multiple imputations followed by averaging of the causal estimates.

Additionally, the paper considers whether the outcome should be included as a predictor in the imputation model.

Hill asserts that while multiple imputation is preferred to other methods of dealing with missing data, in general, there is no a priori reason to prefer one of these techniques over the other. However, there may be reasons to prefer averaging the propensity scores, particularly when using certain matching algorithms. Hill did a a simulation study in the same paper and found that averaging the propensity scores prior to causal inference, when including the outcome in the imputation model produced the best results in terms of mean squared error, and averaging the scores first, but without the outcome in the imputation model, produced the best results in terms of average bias (absolute difference between estimated and true treatment effect). Generally, it is advisable to include the outcome in the imputation model (for example see here).

So it would seem that your method 2 is the way to go.

• I understand method number 2, but I'm at a loss on how to implement it in R. Does anyone have any references to point me to? – sam Apr 12 '15 at 20:52
• R code for both methods is provided in the vignette for the cobalt package entitled "Using cobalt with Complicated Data". You can access it here: CRAN.R-project.org/package=cobalt – Noah Jun 13 '17 at 15:53

There might be a clash of two paradigms. Multiple imputation is a heavily model-based Bayesian solution: the concept of the proper imputation essentially states that you need to sample from the well-defined posterior distribution of the data, otherwise you are screwed. Propensity score matching, on the other hand, is a semi-parametric procedure: once you have computed your propensity score (no matter how, you could've used a kernel density estimate, not necessarily a logit model), you can do the rest by simply taking the differences between the treated and non-treated observations with the same propensity score, which is kinda non-parametric now, as there is no model left that controls for other covariates. I don't feel good about the discontinuities introduced by the literal implementation of matching (find the control with the closest possible value of the propensity score, and ignore the rest; Abadie and Imbens (2008) discussed that it makes it impossible to actually get the standard errors right in some of the matching situations). I would give more trust to the smoother approaches like weighting by the inverse propensity. My favorite reference on this is "Mostly Harmless Econometrics", subtitled "An Empiricist Companion", and aimed at economists, but I think this book should be a required reading for other social scientists, most biostatisticians, and non-bio statisticians as well so that they know how other disciplines approach data analysis.

At any rate, using only one out of 15 simulated complete data lines per observation is equivalent to a single imputation. As a result, you lose efficiency compared to all 15 completed data sets, and you can't estimate the standard errors properly. Looks like a deficient procedure to me, from any angle.

Of course, we happily sweep under the carpet the assumption that both the multiple imputation model and the propensity model are correct in the sense of having all the right variables in all the right functional forms. There is little way to check that (although I'd be happy to hear otherwise about diagnostic measures for both of these methods).

• (+1) In particular for I don't feel good about the discontinuities introduced by the literal implementation of matching (find the control with the closest possible value of the propensity score, and ignore the rest). Propensity scoring has always struck me as quite a rough procedure anyways. – cardinal Sep 14 '12 at 14:27
• @cardinal, see update. – StasK Sep 14 '12 at 16:31
• I've actually seen more criticism of IPTW than I have of matching by other methods (I will need to read up). See Weighting regressions by propensity scores (Freedman & Berk, 2008), and for an applied example see Bjerk, 2009. I'm not quite sure why you recommend Harmless Econometrics in response here, but it is a good recommendation for anyone interested in observational studies none-the-less. – Andy W Sep 14 '12 at 17:08
• @Andy, the Freedman & Berk piece seems to deal with a much simpler situation when you can model everything in a logistic regression. My understanding is that the methods like PSM are applied in much messier situations when you have many more covariates, and you don't trust the model well enough to assume it is correctly specified. They noticed that the situation was favorable for weighting, but I think it was favorable for the model as compared to other possible methods. – StasK Sep 14 '12 at 19:18
• Because your data are not i.i.d., and the great maximum likelihood theorem about the equality of the inverse Hessian and the outer product of the gradient no longer holds, and neither of them is a consistent estimate of variances. One needs to use the sandwich variance estimator, aka linearization estimator in survey statistics, aka White robust estimator in econometrics. – StasK Sep 16 '12 at 21:04

I can't really speak to the theoretical aspects of the question, but I'll give my experience using PS/IPTW models and multiple imputation.

1. I've never heard of someone using multiply imputed data sets and random sampling to build a single data set. That doesn't necessarily mean it's wrong but it's a strange approach to use. The data set also isn't big enough that you'd need to get creative to get around running 3-5 models instead of just one to save time and computation.
2. Rubin's rule and the pooling method is a pretty general tool. Given the pooled, multiply imputed result can be calculated using only the variance and estimates, there's no reason I can see that it couldn't be used for your project - creating the imputed data, performing the analysis on each set, and then pooling. It's what I've done, it's what I've seen done, and unless you have a specific justification not to do it, I can't really see a reason to go with something more exotic - especially if you don't understand what's going on with the method.
• +1 This is a question that is difficult to provide a good answer to as it seems to be such a highly specialized paper. But in addition to claiming to lose the bounty on a previous similar question, the OP added a question begging for solutions that was migrated to meta. I made similar comments to yours in my answer there. I am particularly dubious about regarding sample from the multiply imputed set of data. – Michael Chernick Sep 14 '12 at 11:01
• Thanks ! Do you have any references for where method 2 has been used ? – Joe King Sep 15 '12 at 6:31
• @JoeKing Sadly, not off the top of my head. – Fomite Sep 15 '12 at 6:36