5
$\begingroup$

I'm using propensity score matching to estimate causal treatment effects. I have been concerned about diagnostic metrics for my propensity score model.

However... when I look at the literature, no one seems to report this sort of result. I have even seen a few talks where the speaker claimed (without proof) that performance of the propensity score model is not important.

Can someone please give me some insight about this. Is PS model performance important? If so... are there general guidelines (eg ppv >.9)? If performance is not important... why not? If it is .. why is it seemingly completely neglected in publications.

TIA.

$\endgroup$
0

1 Answer 1

5
$\begingroup$

It is true that propensity score models should not be evaluated based on their predictive ability but rather on their potential to produce balance in the matched dataset. There are multiple philosophies about constructing a propensity score model, which I describe here. In short, you can attempt to construct a propensity score model that approximates the true propensity score model well, thereby ideally imbuing the estimated propensity score with the large-sample balancing properties of the true propensity score, or you can construct a propensity score model that attempts to minimize imbalance in the sample at hand, since attaining balance is the point of using matching methods in the first place. This second philosophy is described in more detail by Ho, Imai, King, and Stuart (2007).

That said, there have been a few studies on this matter. First are a few studies demonstrating that the C-statistic, a measure of the predictive accuracy of a propensity score model, is a poor measure of the potential of a propensity score model to yield an unbiased estimate (Weitzen et al., 2005; Westreich et al., 2011). Another is a study that directly compares propensity score models that seek balance and those that seek predictive accuracy; these methods differ only in how a tuning parameter is chosen (Griffin et al., 2017). They find that optimizing balance yields improved performance over optimizing predictive accuracy.

In my experience, well-fitting propensity scores (i.e., those that seek to optimize prediction) do tend to do well at balancing, but the balance they yield is the standard upon which they should be judged. You can test this yourself using the WeightIt package, which estimates propensity score weights to balance covariates in observational studies. When using the "gbm" method to estimate propensity scores using generalized boosted modeling, you can choose whether you want the tuning parameter to be chosen based on minimizing imbalance or minimizing prediction error with cross-validation. When using the "super" method to estimate propensity scores using SuperLearner (i.e., stacking), you can choose whether you want the predictions to be stacked based on the stacked propensity score's predictive performance (i.e., minimizing the residual sum of squares) or based on its ability to balance the covariates. You will tend to find, of course, that balance-seeking methods achieve balance better, and better balance yields less biased estimates.

See Ali et al. (2015) for guidelines on reporting the quality of propensity scores. They specifically address the weaknesses of the C-statistic of the propensity score and compare it to balance-based measures of propensity score quality. One interesting application of the C-statistic in assessing the quality of a propensity score model is to compute in after a logistic regression in the matched sample; ideally, if there are no differences between the treatment groups, this model should have no predictive accuracy, so a C-statistic indicating this can indicate good balance. This application is known as the "post-matching" C-statistic and was reviewed by Franklin et al. (2014).


Ali, M. S., Groenwold, R. H. H., Belitser, S. V., Pestman, W. R., Hoes, A. W., Roes, K. C. B., Boer, A. de, & Klungel, O. H. (2015). Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal: A systematic review. Journal of Clinical Epidemiology, 68(2), 122–131. https://doi.org/10.1016/j.jclinepi.2014.08.011

Franklin, J. M., Rassen, J. A., Ackermann, D., Bartels, D. B., & Schneeweiss, S. (2014). Metrics for covariate balance in cohort studies of causal effects. Statistics in Medicine, 33(10), 1685–1699. https://doi.org/10.1002/sim.6058

Griffin, B. A., McCaffrey, D. F., Almirall, D., Burgette, L. F., & Setodji, C. M. (2017). Chasing Balance and Other Recommendations for Improving Nonparametric Propensity Score Models. Journal of Causal Inference, 5(2). https://doi.org/10.1515/jci-2015-0026

Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference. Political Analysis, 15(3), 199–236. https://doi.org/10.1093/pan/mpl013

Weitzen, S., Lapane, K. L., Toledano, A. Y., Hume, A. L., & Mor, V. (2005). Weaknesses of goodness-of-fit tests for evaluating propensity score models: The case of the omitted confounder. Pharmacoepidemiology and Drug Safety, 14(4), 227–238. https://doi.org/10.1002/pds.986

Westreich, D., Cole, S. R., Funk, M. J., Brookhart, M. A., & Stürmer, T. (2011). The role of the c-statistic in variable selection for propensity score models. Pharmacoepidemiology and Drug Safety, 20(3), 317–320. https://doi.org/10.1002/pds.2074

$\endgroup$
2
  • $\begingroup$ whoops.... put my comment in the wrong spot and can't seem to change on my phone Thanks for the clear explanation, link, and references. It makes sense to me, and I do understand that achieving balance is the goal. I suppose that if you somehow knew the TRUE model (i. e. there was some explicit and perfectly followed rule for assigning treatment) then an accurate model would also achieve balance. Since we (generally) can never know the rule, or there isn't one, balance is all we can target $\endgroup$
    – MikeS
    Mar 21, 2020 at 13:23
  • $\begingroup$ Thank you very much for the helpful comment. It helped me a lot. $\endgroup$ Nov 26, 2020 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.