I am using propensity score matching to balance covariates between treatment and control samples. The trouble is that the region of common support between the two groups is limited. Primarily, the problem is that the range of values for some of the covariates in the treatment sample is far wider than what is available in the control, even though I have about 25x more control observations than treatment observations.

I can get good covariate balance if I restrict matching to the region of common support, but that means dropping 1/5th of the treatment observations, and the remaining treatment sample is then very dissimilar to the original sample (i.e. probably no longer representative of the general treated population, so it may not be possible to generalize the treatment effect estimate). I have tried multiple different matching methods that does not involve trimming the treatment sample to the region of common support, but then it seems just not possible to get those problematic covariates to within the recommended bounds for good balance.

The question is, what method would under these circumstances give the best estimate of the ATT? Would including the covariates in the model estimating the treatment effect not even out the remaining imbalances in the comparison groups and give a more relevant estimate of the ATT than a perfect match that is not representative of the full range of treatment observations?


This is a great question and one I wish was asked more by research applying propensity/matching methods. In this case, there is a tradeoff between bias due to imbalance (i.e., not discarding treated units but not finding good matches for them) and bias due to distortion of the target population (i.e., discarding treated units to find good matches), or what Rosenbaum and Rubin call "bias due to incomplete matching" (Rosenbaum & Rubin, 1985); this issue is discussed by Wang (2021). One way around this is to rely on extrapolation, i.e., to use a model to extend the information given in the control group to the covariate space of the treated group; of course, this requires assumptions that are likely to be false, leaving you with bias due to extrapolation (which is a kind of model misspecification) (King & Zeng, 2006).

If you are not attached to a target population, though, then discarding treated units can be seen as "moving the goalposts" (Crump et al., 2006), i.e., changing the focus of the analysis to achieve balance in some sample, not necessarily in a representative sample. This approach would be justified in a number of cases, described by one of my favorite papers, Mao, Li, and Greene (2018). If the original sample is not representative of a meaningful population, then the new target population implied by dropping treated units is just as arbitrary as the original target population, and there is no harm done. If you are interested in treatment effect discovery, i.e., discovering whether a treatment effect exists for some group, rather than for a specific group, it again doesn't matter what the target population is. I have addressed this very issue in a question about matching with calipers here.

The question in your title is, "which is more important?" Unfortunately, that is a tough question to answer because it relies on quantities inherently unknowable. For example, if the treatment effect was the same for all units, it wouldn't matter what the target population was, and you could focus all your efforts on achieving balance to accurately estimate that treatment effect. Of course, that is a very strict assumption to make. If you think you can model your way out of this, i.e., by using a well-specified regression model that accurately captures the relationship between the confounders and outcome, then balance isn't so important because the bias due to confounding will be accounted for by the model and you won't have to drop any treated units to estimate the effect. Of course, as I mentioned earlier, it is probably unlikely that any specific model will fully eliminate confounding, especially when it has to extrapolate to the area outside of common support.

I would venture that your original target population is not so critical to you, especially since the ATT generalizes to a fuzzy population anyway (i.e., defined by a latent propensity to receive treatment rather than by a specific a priori specified covariate distribution), in which case I would recommend you use a method that allows you to robustly estimate a treatment effect even if you have to discard some treated units to do so. Because I don't know exactly what your data look like, it's hard to make a firm recommendation (and you should try several methods, too), but I would recommend either cardinality matching, which is particularly effective in the absence of overlap (Visconti & Zubizarreta, 2018), or a more basic matching method (like propensity score or Mahalanobis distance matching) with calipers set on specific prognostically important covariates. You can use a simple regression of the outcome on the covariates, treatment, and their interaction to reduce any bias due to remaining imbalance and improve precision in the estimate, and this model will not succumb to extrapolation because you will already have restricted the range of the sample by matching (Ho, Imai, King, & Stuart, 2007).

Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2006). Moving the goalposts: Addressing limited overlap in estimation of average treatment effects by changing the estimand. 2347. http://hdl.handle.net/10419/34048

Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference. Political Analysis, 15(3), 199–236. https://doi.org/10.1093/pan/mpl013

King, G., & Zeng, L. (2006). The dangers of extreme counterfactuals. Political Analysis, 14(2), 131–159. https://doi.org/10.1093/pan/mpj004

Mao, H., Li, L., & Greene, T. (2018). Propensity score weighting analysis and treatment effect discovery. Statistical Methods in Medical Research, 096228021878117. https://doi.org/10.1177/0962280218781171

Rosenbaum, P. R., & Rubin, D. B. (1985). The Bias Due to Incomplete Matching. Biometrics, 41(1), 103–116. JSTOR. https://doi.org/10.2307/2530647

Visconti, G., & Zubizarreta, J. R. (2018). Handling Limited Overlap in Observational Studies with Cardinality Matching. Observational Studies, 5, 33.

Wang, J. (2021). To use or not to use propensity score matching? Pharmaceutical Statistics, 20(1), 15–24. https://doi.org/10.1002/pst.2051


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.