Propensity score matching vs non-parametric regression

Question

I am trying to understand the benefit of propensity matching over non-parametric regression for causal inference from non-experimental data.

As background: the way I understand it, parametric regressions are generally a poor choice for causal inference when selection bias exists. One can try to create a model that takes into account the different baselines between treatment and control groups, but she/he will be be extremely vulnerable to model misspecification.*

Matching solves this issue by making the two datasets look "alike." The exact meaning of alike depends on the matching algorithm used, but all the algorithms strive to eradicate model dependence, with the most common types being stratified and weighted propensity matching. The mathematics for these algorithms are well-developed, and in particular, we know how to extract error bars and statistical significance.

What I'm struggling with is why this is superior to just using a non-parametric regression like a decision tree or random forest, which are also designed to prevent model misspecification. After creating the forest, one could run individuals through it assuming treatment or no treatment, and call the difference the estimated treatment effect for that individual. My first guess is that extracting significance, which is critical to causal inference, from trees is difficult, but it seems that statisticians have made strides in that regard over the last decade or so. To be clear, I am not asking about using a tree to develop the propensity scores, but to use one instead of propensity matching.

To help kick off the conversation, I've developed five hypotheses for why matching is preferred to non-parametric regression, but haven't been able to find anything proving or disproving any:

1. Empirical research demonstrates that stratified or weighted propensity matching (the most common types) yield results closer to causal experiments than non-parametric regressions like trees.

2. Though it is possible to extract significance from non-parametric regressions like random forests, the math isn't settled, or the notion of "significance" for a decision tree variable doesn't map precisely to the notion of "one minus the odds of a type I error."

3. Though it is possible to extract significance from non-parametric regressions, the code is difficult to write.

4. Model misspecification actually is an issue for decision trees due to the tuning required to run them. Empirical observation has demonstrated this is more of an issue for decision trees than it is for matching algorithms.

5. We don't actually know a lot about whether we can use non-parametric regressions for causal inference, but we do know that matching works, so it there's no reason to reinvent the wheel.

*As detailed in the first ten minutes of this wonderfully intuitive Youtube: https://www.youtube.com/watch?v=rBv39pK1iEs

Answer 1

This is a great question and one for which there is no single answer, so I won't attempt to give one to be comprehensive. I'll mention a few topics that might satisfy some of your curiosity and point you to some interesting studies seeking to address the question you asked.

The method you described of training a random forest and then producing predictions under the treatment and under control is a well-established and somewhat popular method called g-computation. The bootstrap is often used to estimate confidence intervals for effects estimated with g-computation. A recently popular method of g-computation uses Bayesian additive regression trees (BART) as the model; it has proven to be very successful and straightforward to use because it does not require parameter tuning. Inference is straightforward because it produces a Bayesian posterior from which credible intervals can be computed and interpreted as confidence intervals.

There is a class of methods known as "doubly-robust" methods that involve estimating both an outcome model and a propensity score model and combining them. A benefit of these methods is that the estimate is consistent (i.e., unbiased in large samples) if either the propensity score model or outcome model is correct, and often inference is straightforward with these methods. Examples of doubly-robust methods include augmented inverse probability weighting (AIPW), targeted minimum loss-based estimation (TMLE), g-computation in propensity score-matched samples, and BART with the propensity score as an additional covariate. These methods are gaining popularity and are widely discussed in the statistics literature. They combine the best of both outcome modeling and treatment modeling.

That said, many researchers prefer to only use matching and other treatment model-focused methods like weighting. I'll provide a short list of some of the primary motivations I've seen:

• Matching methods can be more robust to model misspecification, making their estimates more trustworthy
• Matching and weighting involve the assessment and reporting of covariate balance, which provides evidence to the reader that the method has plausibly reduced all the bias due to the measured covariates (this cannot be done with outcome regression)
• With matching and weighting, one can try many different methods without estimating a treatment effect to find the one that is going to be the most trustworthy. With outcome modeling, you only get one chance, or else you succumb to capitalization on chance and the potential to try many models until the desired effect is found
• Matching methods are easier to understand and explain to lay audiences
• Matching and weighting are agnostic to the outcome type, so they can be used in larger models or for outcome types for which g-computation is less straightforward, like survival outcomes
• Matching and weighting methods are sometimes found to be less biased than g-computation in simulations
• Matching and weighting are more transparent and customizable; it is easier to incorporate substantive expertise into the way certain variables are prioritized
• Matching and weighting do not involve extrapolation beyond the region of common support

Hopefully that list gets you started in trying to understand this choice. Unfortunately the question of "should I use matching or g-computation for my data?" is basically equivalent to "what is the correct model for my data?" which is an eternal mystery. The "correct" answer for any given dataset is unknown, and some methods may be better suited for different kinds of datasets based on qualities that are unobservable.

To specifically address your hypotheses:

1. Yes, sometimes, though combinations of both tend to do best.
2. Yes-ish; bootstrapping is often used, but not necessarily always valid. For some methods, we can use Bayesian to help. G-computation is not too hard to implement nonparametrically but it often has to be manually programmed.
3. Same as 2).
4. Absolutely yes. Just because a method is flexible doesn't mean it will always get the answer right. There is an inherent bias-variance tradeoff that must be managed with all methods. BART tends to do better than other machine learning methods because of how it balances flexibility with precision.
5. Not really; we know a lot about how to use them, but we know a lot about how they can be improved and using doubly robust methods in many cases dramatically improves their performance.

Answer 2

I found this set of lecture notes to be quite helpful: https://mlhcmit.github.io/slides/lecture15.pdf

There are two common approaches for counterfactual inference, Propensity Scores, and Covariate adjustment.

For Covariate Adjustment you explicitly model the relationship between treatment, confounders, and outcome. Obviously there are lots of options on how to model the relationship, from linear regression, to more advanced techniques, for example random forests and deep learning

To be honest, I'm not sure of why to prefer one approach vs. the other, one thought is perhaps if you're not confident about how to model the causal relationship, or if you've captured all the confounders, but you're able to predict treatment well, then you might favor propsensity score matching?