1
$\begingroup$

I'm working on a project with healthcare data where episodes of care in the treatment and control groups must be matched to estimate average treatment effect (ATE).

I have several hundred covariates including age, sex, and various utilization and health risk factors (most of which are sparse binary variables).

While my go-to model has been propensity score matching, recent research by Gary King, Richard Nielsen, and others have recommended using exact matching (or coarsened exact matching).

Question: Is it appropriate to use exact matching on a subset of the covariates, followed by multiple regression on the remaining variables in the matched data in order to estimate the ATE? The subset of covariates could be selected by choosing the largest effect sizes from a simple regression on the outcome variable.

$\endgroup$
1
$\begingroup$

The King & Nielsen (2016) paper is misleading. It has not been peer-reviewed, and it makes a claim about the results of a testable assumption that you can assess in your own data set (i.e., whether propensity score matching produces balance). In addition, there has been some work to debunk the paper, and several instances in which coarsened exact matching does not perform well. The paper has since been peer-reviewed and accepted. The claims it makes are still empirically verifiable, though, so there is no reason to categorically avoid using propensity score matching. The paper makes testable predictions, and you can assess whether those predictions are realized in your data. The prediction is that propensity score matching will not yield balance; you can simply assess balance in your data after matching.

Update (1/1/20): A paper examining the validity of King & Nielsen (2019) in pharmacoepidemiology is Ripollone et al. (2018). They find that the propensity score paradox does occur, but far beyond recommended and common practices for propensity score matching. In applications, propensity score matching is effective at achieving bias and lowering bias. The same group (Ripollone et al. 2019), who maybe have a bone to pick with King, also evaluate the performance of CEM and find that it yields extremely high error in the effect estimates compared to propensity score matching.

The question of how to deal with high-dimensional covariates in causal inference is really hot right now, and there are several modern techniques that have been developed that you should consider before propensity score matching and regression. Matching and regression are some of the earliest causal inference techniques and there has been so much advancement upon these methods that really no one should be using their basic forms. Here are some recommendations for causal inference tools for high-dimensional data:

Targeted Minimum Loss-Based Estimation (TMLE) - TMLE is a doubly-robust effect estimator that relies on machine learning and regression to remove confounding without making functional form assumptions about the treatment or outcome model. There is a version called "Collaborative" TMLE (CTMLE), which specifically addresses the problem of high-dimensional covariates. TMLE has been shown to do very well in simulations and in a recent causal inference competition (Dorie et a., 2019). It's very easy to implement and there is an easy-to-use R package (TMLE) to do it. It is becoming the gold standard in causal inference. See Schuler & Rose (2017) for an introduction.

Bayesian Additive Regression Trees (BART) - BART is a machine learning method that uses Bayesian components both to yield good performance and inference. It works like a flexible outcome regression model, but you can include the propensity score (potentially also estimated using BART) to increase its robustness and performance. Because it only prioritizes covariates that are predictive of the outcome, it automatically selects the relevant variables from a potentially long list, and therefore is effective in high dimensions. It has also been shown to have great performance and to have done will in the causal inference competition, and there is also an easy-to-use R package (bartCause) to implement it. See Hill (2011) for an introduction.

Group Lasso with Doubly Robust Estimation (GLIDER) - GLIDER is a double-robust propensity score weighting + regression estimator that is especially useful in high dimensions. It uses lasso to select the right covariates that predict both the outcome and the propensity score. It uses an adaptive lasso, which means the coefficients are asymptotically unbiased. It is straightforward to include many transformations of variables to account for potential nonlinearities; if they aren't useful in the model, they are lasso'ed out. See Koch, Vock, & Wolfson (2018) for an introduction.

Hopefully that should get you started. Matching and regression do not appear appropriate to me in this case, and there are several better-performing methods that would suit your goals. You should consult with a biostatistician rather than attempt to implement out-of-date methods.


Dorie, V., Hill, J., Shalit, U., Scott, M., & Cervone, D. (2019). Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition. Statistical Science, 34(1), 43–68. https://doi.org/10.1214/18-STS667

Hill, J. L. (2011). Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics, 20(1), 217–240. https://doi.org/10.1198/jcgs.2010.08162

King, G., & Nielsen, R. (2016). Why propensity scores should not be used for matching. Retrieved from http://www.polmeth.wustl.edu/files/polmeth/psnot4.pdf

King, G., & Nielsen, R. (2019). Why Propensity Scores Should Not Be Used for Matching. Political Analysis, 1–20. https://doi.org/10.1017/pan.2019.11

Koch, B., Vock, D. M., & Wolfson, J. (2018). Covariate selection with group lasso and doubly robust estimation of causal effects. Biometrics, 74(1), 8–17. https://doi.org/10.1111/biom.12736

Ripollone, J. E., Huybrechts, K. F., Rothman, K. J., Ferguson, R. E., & Franklin, J. M. (2018). Implications of the Propensity Score Matching Paradox in Pharmacoepidemiology. American Journal of Epidemiology, 187(9), 1951–1961. https://doi.org/10.1093/aje/kwy078

Ripollone, J. E., Huybrechts, K. F., Rothman, K. J., Ferguson, R. E., & Franklin, J. M. (2019). Evaluating the Utility of Coarsened Exact Matching for Pharmacoepidemiology using Real and Simulated Claims Data. American Journal of Epidemiology, kwz268. https://doi.org/10.1093/aje/kwz268

Schuler, M. S., & Rose, S. (2017). Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies. American Journal of Epidemiology, 185(1), 65–73. https://doi.org/10.1093/aje/kww165

$\endgroup$
  • $\begingroup$ Thank you, very helpful. I'll take a look at the papers you've cited - do you have any references for refutations of King & Nielsen's paper? Are propensity score matching and exact matching completely out-of-date, or just two of many approaches that ought to be tested & compared with more recent methods? Another question (which perhaps I should post separately) is how to compare a technique like exact matching with some of the machine learning models you've mentioned. For example, CEM and PSM can be compared via balance diagnostics on the covariates, but not with regression or ML models. $\endgroup$ – RobertF Jul 2 '19 at 16:17
  • 1
    $\begingroup$ There was a paper that attempted to refute K&N, but I can't find it anymore after some searching, but the success of PSM in many simulations is enough for me, I think. PSM is not completely out-of-date, but there have been many improvements on it that you should consider before going straight for it. Exact matching is still the gold standard and always will be, but it tends to be impossible in mot circumstances. CEM would be ideal too, but many data sets don't support it (see, e.g., Zubizarreta et al. (2014)). $\endgroup$ – Noah Jul 2 '19 at 16:40
  • 1
    $\begingroup$ I do know about FLAME, but it's primarily for massive data sets (i.e., that exist in databases) rather than typical samples. It's not ready for prime-time. The comparison of performance between the methods is tough and is a reason to prefer matching and weighting over regression-based, but you can also rely on the theoretical characteristics of the methods (e.g., that they tend to do well in arbitrary simulations). You could make your own simulation based on your data and see under what circumstances one method does better. $\endgroup$ – Noah Jul 2 '19 at 16:43
  • 1
    $\begingroup$ PSM with reduction of sample size due to the matching algorithm is likely to be statistically inefficient (higher standard error and lower power of treatment effect). $\endgroup$ – Frank Harrell Jul 5 '19 at 20:04
  • 1
    $\begingroup$ We teach both traditional PSM, regression adjustment using PS, inverse probability weighting, and efficient match-while-randomizing methods. The latter increases efficiency of treatment comparisons. $\endgroup$ – Frank Harrell Jul 29 '19 at 12:48
2
$\begingroup$

It's important to answer the question "why are we using matching in this study?" There are lots of good reasons to use matching, but wanting to estimate the ATE isn't one of them. Most matching methods create a cohort that is excellent for estimating the effect of exposure but whose covariate distribution is no longer identical to the source population. That is, the matching methods drop or down-weight regions of poor common support in the covariate space.

If you really want to estimate the ATE in this setting, consider just fitting a regression model on the source population (possibly using a relaxed lasso to help with the numerous sparse binary variables) and then use the model to estimate your ATE. You can use the model to calculate the predicted the outcomes for everyone in the source population (once under treatment and once under control). You now have predicted (Y0, Y1) pairs for everyone, and calculating the ATE is straightforward. The estimate will only be as good as the model. For example, you may have to assume additive effects for the binary variables that are being estimated largely from one of the exposures alone. You may be doing extensive extrapolation into regions where there is little common support. That depends on your particular dataset.

One question to ask is "how important is it to estimate the ATE for this study?" Are you really interested in the average effect over the source population or are you more interested in a persuasive study design that yields a robust estimate of the treatment effect over a well defined study population? If it's the latter, the matching methods can be helpful.

$\endgroup$
  • $\begingroup$ The goal is estimating how effective the treatment program is at reducing medical costs. Using regression by itself is an option, although we then sacrifice the ability to control for complex interactions between covariates. Lasso is certainly a powerful tool with high-dimensional data, but would it be appropriate to use if the aim is causal inference rather than prediction? Lasso models introduce bias to minimize MSE, and in this study we prefer unbiased estimates. $\endgroup$ – RobertF Jul 1 '19 at 18:37
  • 1
    $\begingroup$ (1) That's a good argument for the ATE. (2) You can specify interactions in the model. But if you're dealing with a sparse covariate matrix like you described, you may not be able to control for complex interactions regardless. That may be a limitation of your dataset. (3) The link between the prediction model and the causal inference you want to do is in the calculation of the ATE. When you use the conditional estimates from the model to get an average effect over a given distribution of covariates you are connecting them. ... $\endgroup$ – Robert Alan Greevy Jr PhD Jul 1 '19 at 21:10
  • 1
    $\begingroup$ (4) Accepting some bias is likely a limitation of your dataset. You suggested you can't control for every covariate and have to focus on the most impactful ones. So you're probably going to have to live with the possibility of some residual confounding. The relaxed lasso can help with your bias concern somewhat by controlling the penalty for model selection and shrinkage separately. $\endgroup$ – Robert Alan Greevy Jr PhD Jul 1 '19 at 21:19
  • 1
    $\begingroup$ 1) Almost. You can use one penalty that impacts how many coefficients get shrunk to 0, i.e. how much gets dropped in the model selection stage. You can use a different penalty to decide how much the non-zero coefficients get shrunk, including not shrinking them at all. So you won't call the estimate unbiased because you did shrink some coefficients to 0 in the model selection stage. But once you have the non-zero coefficients, you can choose to re-estimate them without shrinkage in the estimation stage. $\endgroup$ – Robert Alan Greevy Jr PhD Jul 5 '19 at 19:13
  • 1
    $\begingroup$ 2) Exactly. If you have interactions with treatment and/or if you use a transformation of medical costs so you are fitting a noncollapsible model, you estimate the subject specific effects by predicting the cost for every subject under each treatment and taking the difference. Then you estimate the ATE by averaging the subject specific effects. $\endgroup$ – Robert Alan Greevy Jr PhD Jul 5 '19 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.