5
$\begingroup$

When used successfully to balance the covariate distributions in the treatment and control groups, is it safe to say Coarsened Exact Matching (CEM) is the best method for finding an unbiased estimate of the Average Treatment Effect (ATE)? I'm not aware of any research comparing CEM with more recent methodologies.

Perhaps one limitation with CEM is selecting the correct subset of matching variables if there are hundreds or thousands to choose from. One strategy is to select the top $k$ variables for CEM by ranking their coefficients from an OLS regression, but this isn't guaranteed to find the optimal subset.

$\endgroup$

1 Answer 1

10
$\begingroup$

CEM does not allow you to estimate the ATE. This is because the matched units in each treatment group will not resemble the overall sample. If no treated units are unmatched, you can estimate the average treatment effect on the treated (ATT). If any treated units are discarded, the estimand is an average treatment effect, but not for a specific pre-defined population; it's the average treatment effect in the matched sample (ATM).

The best method for estimating the ATM is exact matching. If exact matching is performed on the set of variables sufficient to remove confounding (I'll call these confounders), regardless of the form of the outcome model, the treatment effect will be unbiased, even in finite samples. This is because the samples will be exactly balanced on all confounders and their entire joint distribution. Generally, if there are continuous confounders or many confounders relative to the size of the control pool, exact matching will be impossible.

This phenomenon is known as the curse of dimensionality and is why propensity score matching became popular; rather than exact matching on every confounder, Rosenbaum & Rubin (1983) proved that exact matching on the true propensity score also balanced the joint distribution of confounders in large samples and therefore yields asymptotically unbiased and consistent estimates. A problem with the common implementation of propensity score matching is that it departs from the theoretical results in several ways: it is used in small samples, it uses an imperfect estimate of the propensity score, and it is only approximately matched. King & Nielsen (2019) also demonstrated in their infamous paper that propensity score matching as commonly implemented will fail to extract a randomized block experiment from a confounded sample, instead extracting only a randomized experiment, which is less efficient and therefore more model-dependent. All that said, propensity score matching does tend to work fairly well in practice if done right, as demonstrated by extensive simulation evidence, though there is also much simulation evidence demonstrating how its common uses can perform extremely poorly.

The problem with propensity score matching in finite samples is that when the propensity score is not known, it must be estimated, and the assessment of its correct specification relies on balance checking. The point of propensity score matching is to attain balance anyway, but ideally, propensity score matching yields balance on the joint distribution of all the confounders. Unfortunately, it's hard to assess balance on the joint distribution, though there have been attempts. Instead, we typically assess balance only on the means of each confounder individually. Simulations have shown that this can be an effective strategy, however (Franklin et al., 2014).

The problem is that it requires assumptions about the form of the outcome model. The whole point of matching is to avoid these assumptions; otherwise, if they were known, you could just model the outcome and your estimate would be far more precise. The presumed logic of balance checking for propensity score matching, then, is that if balance is achieved on the terms one checked balance for, it is also achieved in the joint distribution of confounders, so one doesn't need to make assumptions about the form of the outcome model. If you are skeptical of this logic, then you have to either know the form of the outcome model or know the form of the propensity score model and have very close matches.

CEM aims to avoid these problems by capitalizing on the strength of exact matching without succumbing to the curse of dimensionality. It does this by coarsening continuous variables and combining levels of categorical variables. It's more likely that you can find exact matches on the coarsened confounders than in the original confounders. Another selling point of CEM is that you get to control how balanced the sample is by adjusting the degree of coarsening; with no coarsening, you have exact matching and therefore exact balance on the joint distribution of confounders (if the data supports it), and with extreme coarsening you have individuals matched that are not very similar to each other, and therefore less balance. That's why Iacus et al. (2011) titled their paper "Causal Inference Without Balance Checking: Coarsened Exact Matching."

CEM unfortunately still succumbs to the curse of dimensionality in most samples because unless the coarsening is extreme, it's rare to find exact matches for every treated unit, so many treated units are discarded. In the remaining matched sample, however, approximate balance is achieved on the joint distribution of confounders, so the effect estimate will be approximately unbiased regardless of the form of the outcome model. CEM will be useful in the following scenario:

  1. A large control pool with strong overlap with the treated units
  2. Several continuous confounders
  3. The effect estimate doesn't have to generalize to a target population or assumed to be the same for all units
  4. The outcome model is highly nonlinear in the confounders and depends on their interactions

All of these must be true for CEM to be of value; if they are true, CEM is undoubtedly the best matching method for the reasons described in Iacus et al. (2011). If any of them are false, there is a better method out there. Below I'll discuss some alternatives and their strengths over CEM.

  • Genetic matching (Diamond & Sekhon, 2013) - recovers randomized block experiments; guarantees balance as the user defines it; doesn't have to discard treated units; in the Matching R package
  • Cardinality matching (Zubizarreta et al., 2014) - balance constraints can be specified without requiring exact balance on the joint distributions of confounders or their coarsened versions; in the designmatch R package
  • ATO weighting (Li & Thomas, 2018) - most precise weighted estimate possible, guarantees exact moment balance on each covariate (and many moments can be specified to capture the joint distribution); in the WeightIt R package
  • BART (Hill, 2011)/TMLE (van der Laan, 2010) - extremely flexible without assumptions on the outcome or treatment model and without discarding any units; in the bartCause and TMLE R packages

In the case you described, where you have many potential confounders to match on, there is FLAME (Wang et al., 2019), available in FLAME, and its successors.

I'm sorry this was so much, but this is a topic that deserves discussion and consideration. I spend my days thinking about it (actually; it's my line of research). Everything boils down to whether you want to make certain assumptions and how you can manage the bias-variance tradeoff given those assumptions. There is no right answer.


Diamond, A., & Sekhon, J. S. (2013). Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies. Review of Economics and Statistics, 95(3), 932–945. https://doi.org/10.1162/REST_a_00318

Franklin, J. M., Rassen, J. A., Ackermann, D., Bartels, D. B., & Schneeweiss, S. (2014). Metrics for covariate balance in cohort studies of causal effects. Statistics in Medicine, 33(10), 1685–1699. https://doi.org/10.1002/sim.6058

Iacus, S. M., King, G., & Porro, G. (2011). Causal Inference without Balance Checking: Coarsened Exact Matching. Political Analysis, mpr013. https://doi.org/10.1093/pan/mpr013

Hill, J. L. (2011). Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics, 20(1), 217–240. https://doi.org/10.1198/jcgs.2010.08162

King, G., & Nielsen, R. (2019). Why Propensity Scores Should Not Be Used for Matching. Political Analysis, 1–20. https://doi.org/10.1017/pan.2019.11

Li, F., & Thomas, L. E. (2018). Addressing Extreme Propensity Scores via the Overlap Weights. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwy201

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

van der Laan, M. J. (2010). Targeted Maximum Likelihood Based Causal Inference: Part I. The International Journal of Biostatistics, 6(2). https://doi.org/10.2202/1557-4679.1211

Wang, T., Morucci, M., Awan, M. U., Liu, Y., Roy, S., Rudin, C., & Volfovsky, A. (2019). FLAME: A Fast Large-scale Almost Matching Exactly Approach to Causal Inference. ArXiv:1707.06315 [Cs, Stat]. http://arxiv.org/abs/1707.06315

Zubizarreta, J. R., Paredes, R. D., & Rosenbaum, P. R. (2014). Matching for balance, pairing for heterogeneity in an observational study of the effectiveness of for-profit and not-for-profit high schools in Chile. The Annals of Applied Statistics, 8(1), 204–231. https://doi.org/10.1214/13-AOAS713

$\endgroup$
7
  • $\begingroup$ This is fantastic, thank you Noah! Now I see one reason why TMLE and CEM haven't been directly compared. At present the tmle package in R doesn't estimate ATT, only ATE, while the cem package estimates ATT. $\endgroup$
    – RobertF
    Apr 25, 2020 at 16:01
  • $\begingroup$ Glad it was helpful. CEM only computes the ATT when no treated units are discarded. The bartCause package allows you to compute the ATT using BART. $\endgroup$
    – Noah
    Apr 25, 2020 at 16:03
  • $\begingroup$ I can see more clearly the limitations of CEM compared to other techniques like TMLE. The insurance claims data I work with has highly imbalanced variable categories for patient's diagnosis & procedure codes and geography. After using CEM to match by sex and age, you're faced with a bewildering number of choices where to match next. You can't just match on diagnosis code, as there are thousands of categories, many with < 10 observations. OLS regression helps to select the top $k$ categories to match on but seems somewhat arbitrary choosing a cutoff this way based on p-values and effect sizes. $\endgroup$
    – RobertF
    Apr 25, 2020 at 16:31
  • 1
    $\begingroup$ Have you looked into using lasso to select the categories to include? Check out Shortreed & Ertefaie (2017). You can use CEM on the variables it's easy to match on and then use a lasso propensity score to match the others. $\endgroup$
    – Noah
    Apr 25, 2020 at 16:36
  • 1
    $\begingroup$ Just came across this interesting pre-publication paper that is quite critical of CEM: papers.ssrn.com/sol3/papers.cfm?abstract_id=3694749 $\endgroup$
    – RobertF
    Jan 20, 2021 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.