How to compare different causal inference methodologies for estimating Average Treatment Effect when true treatment effect is unknown?

I'm comparing various methods for estimating average treatment effects (ATEs) for cost savings in a case-control study on health insurance episode of care data for my employer.

My company currently uses coarsened exact matching (CEM), often with a follow-up regression on matched episodes, to estimate ATEs. I'd like to compare more recently developed techniques that are better suited for high dimensional data, such as Targeted Minimum Loss-Based Estimation (TMLE) and Bayesian Additive Regression Trees (BART).

Is there a way to compare techniques for estimating the Average Treatment Effect without knowing the true ATE in advance?

I'm considering two possibilities:

• Build a data generating process that closely resembles our own data. This could be accomplished by fitting a linear regression model Y ~ treatment + covariate_main_effects + covariate_interaction_effects to the data, where the covariates in would be manually selected. Then define the regression coefficient for the treatment covariate as the true ATE to be estimated, and generate data according to the model, say using the simstudy package. Causal inference methods are then tested on the simulated data. My colleagues are hesitant to use a data generating process, they would prefer to test on our observed claims data. But perhaps this method will allay their fears.
• Instead of using a data generating process, select a subset of variables from the analysis data so that there are no positivity violations (that is, every matching subgroup contains a suitable number of episodes, say at least 5, in each of the treatment and control categories). Then find the true ATE_subset by comparing average episode costs for treatments and controls within each subgroup. The disadvantage of this approach is I'm testing on a small subset of the population, but it would establish a baseline confidence in new causal inference techniques' ability to find the true ATE.

Your first intuition is correct. This is called a plasmode simulation and is a great and frequently used way to compare the performance of estimators for data with covariate and treatment distributions like the one you have. See Franklin et al. (2014) who use this method exactly as you intend. The key, though, is in choosing a model with which to generate the outcome. ideally, it would reflect a data-generating process similar to the one that exists in the real data, but if you knew that, you wouldn't need to estimate treatment effects in the first place. What you can do instead is to simulate data with varying qualities and relationships (e.g., linear, quadratic, piecewise, etc.). This way, if you can demonstrate one method works on all (or most) data-generating scenarios, you have evidence in favor of using it for your dataset.

The 2016 Atlantic Causal Inference Conference data competition did exactly this. The paper about it (Dorie et al, 2019) is very thorough. It explains the many ways they generated the data and what features each data-generating model had that would make it more or less challenging for estimators. The results of the competition were that BART and TMLE dramatically outshone all the other methods, although CEM was not included. This type of simulation study has no less merit than the simulation studies you typically see in statistics papers demonstrating a new method or comparing methods. You can also use the data from this contest to compare your own method. They have an R package described in the article which allows you to simulate the data they simulated and apply your own pet method to it. It's unlikely this data will look much like your own dataset, though, so this alone might not sway your colleagues.

Ask your colleagues why they like CEM so much. It's a generally poorly performing method in that it discards huge amounts of data, the causal effect doesn't correspond to the average treatment effect in any population of interest specified a priori (i.e., not the ATT or ATE), and it relies on many arbitrary decisions by the researcher. I can't think of any study other than the one introducing CEM in which it outperforms another method. One paper that comes to mind is Zubizarreta et al (2014), in which it does so poorly, an effect can't even be estimated from it (i.e., it throws away all the data).

I just noticed you are doing a case-control study rather than a case-cohort study. I'm not a biostatistician so I don't know whether the same methods can be used for both types of study, but maybe someone else can comment on that.

• Thanks again, Noah. My coworkers like CEM because it's relatively transparent (easy to explain), provides more accurate estimation of ATE or ATT if the match is 100%, and better controls for unmeasured confounders by matching subjects on similar characteristics. I'll grant the first point, the other two are debatable. Especially given I have a far from 100% match in the study I'm evaluating since a 3:1 exact match was used. Although I'm agnostic, it may turn out CEM is more accurate than alternatives with our data - if a proper comparison methodology can be devised. Nov 13 '19 at 15:16
• Yes, this is a case-control study. We're not following individuals from the pre- to post-treatment periods in both treatment and control groups, but rather are comparing episodes of care on different individuals. Do you think my 2nd proposed method holds merit, as an alternative to using a DGP? Nov 13 '19 at 15:19
• I don't really understand how that method works. You're estimating a treatment effect in that subset, are you not? And that estimate will have error because of the method you use to estimate it. You can't get free information about the truth from your sample; you have to make an assumption. I don't see how your method will give you a truth to which you can compare potential estimates.
– Noah
Nov 13 '19 at 18:31
• My thinking is that we could obtain an estimate of the "true" ATE and it's variance but only for a subset of say 10 variables. It couldn't be generalized to the entire dataset containing hundreds of variables, but it could serve as a benchmark that TMLE or BART is producing unbiased ATE estimates on "real world data" without using a data generating process. This would give us more confidence to use TMLE, BART, etc. on larger datasets. Nov 13 '19 at 20:58
• I don't know what you mean by a "subset of variables". I understand on a subset of individuals, but the ATE has to marginalize over the entire covariate distribution. I'm also still unclear on how you would get the true ATE for anyone, which is required to serve as a benchmark. I don't really understand your method but I can't see how it could have any epistemic advantages over controlling the data-generating process. Simulation studies are the only way I know of to compare methods in an arbitrary population, and a plasmode simulation will be most effective for your purpose.
– Noah
Nov 14 '19 at 0:38