I'm comparing various methods for estimating average treatment effects (ATEs) for cost savings in a case-control study on health insurance episode of care data for my employer.
My company currently uses coarsened exact matching (CEM), often with a follow-up regression on matched episodes, to estimate ATEs. I'd like to compare more recently developed techniques that are better suited for high dimensional data, such as Targeted Minimum Loss-Based Estimation (TMLE) and Bayesian Additive Regression Trees (BART).
Is there a way to compare techniques for estimating the Average Treatment Effect without knowing the true ATE in advance?
I'm considering two possibilities:
- Build a data generating process that closely resembles our own data. This could be accomplished by fitting a linear regression model Y ~ treatment + covariate_main_effects + covariate_interaction_effects to the data, where the covariates in would be manually selected. Then define the regression coefficient for the treatment covariate as the true ATE to be estimated, and generate data according to the model, say using the
simstudypackage. Causal inference methods are then tested on the simulated data. My colleagues are hesitant to use a data generating process, they would prefer to test on our observed claims data. But perhaps this method will allay their fears.
- Instead of using a data generating process, select a subset of variables from the analysis data so that there are no positivity violations (that is, every matching subgroup contains a suitable number of episodes, say at least 5, in each of the treatment and control categories). Then find the true ATE_subset by comparing average episode costs for treatments and controls within each subgroup. The disadvantage of this approach is I'm testing on a small subset of the population, but it would establish a baseline confidence in new causal inference techniques' ability to find the true ATE.