9
$\begingroup$

I am trying to understand the benefit of propensity matching over non-parametric regression for causal inference from non-experimental data.

As background: the way I understand it, parametric regressions are generally a poor choice for causal inference when selection bias exists. One can try to create a model that takes into account the different baselines between treatment and control groups, but she/he will be be extremely vulnerable to model misspecification.*

Matching solves this issue by making the two datasets look "alike." The exact meaning of alike depends on the matching algorithm used, but all the algorithms strive to eradicate model dependence, with the most common types being stratified and weighted propensity matching. The mathematics for these algorithms are well-developed, and in particular, we know how to extract error bars and statistical significance.

What I'm struggling with is why this is superior to just using a non-parametric regression like a decision tree or random forest, which are also designed to prevent model misspecification. After creating the forest, one could run individuals through it assuming treatment or no treatment, and call the difference the estimated treatment effect for that individual. My first guess is that extracting significance, which is critical to causal inference, from trees is difficult, but it seems that statisticians have made strides in that regard over the last decade or so. To be clear, I am not asking about using a tree to develop the propensity scores, but to use one instead of propensity matching.

To help kick off the conversation, I've developed five hypotheses for why matching is preferred to non-parametric regression, but haven't been able to find anything proving or disproving any:

  1. Empirical research demonstrates that stratified or weighted propensity matching (the most common types) yield results closer to causal experiments than non-parametric regressions like trees.

  2. Though it is possible to extract significance from non-parametric regressions like random forests, the math isn't settled, or the notion of "significance" for a decision tree variable doesn't map precisely to the notion of "one minus the odds of a type I error."

  3. Though it is possible to extract significance from non-parametric regressions, the code is difficult to write.

  4. Model misspecification actually is an issue for decision trees due to the tuning required to run them. Empirical observation has demonstrated this is more of an issue for decision trees than it is for matching algorithms.

  5. We don't actually know a lot about whether we can use non-parametric regressions for causal inference, but we do know that matching works, so it there's no reason to reinvent the wheel.

*As detailed in the first ten minutes of this wonderfully intuitive Youtube: https://www.youtube.com/watch?v=rBv39pK1iEs

$\endgroup$
8
$\begingroup$

This is a great question and one for which there is no single answer, so I won't attempt to give one to be comprehensive. I'll mention a few topics that might satisfy some of your curiosity and point you to some interesting studies seeking to address the question you asked.

The method you described of training a random forest and then producing predictions under the treatment and under control is a well-established and somewhat popular method called g-computation. The bootstrap is often used to estimate confidence intervals for effects estimated with g-computation. A recently popular method of g-computation uses Bayesian additive regression trees (BART) as the model; it has proven to be very successful and straightforward to use because it does not require parameter tuning. Inference is straightforward because it produces a Bayesian posterior from which credible intervals can be computed and interpreted as confidence intervals.

There is a class of methods known as "doubly-robust" methods that involve estimating both an outcome model and a propensity score model and combining them. A benefit of these methods is that the estimate is consistent (i.e., unbiased in large samples) if either the propensity score model or outcome model is correct, and often inference is straightforward with these methods. Examples of doubly-robust methods include augmented inverse probability weighting (AIPW), targeted minimum loss-based estimation (TMLE), g-computation in propensity score-matched samples, and BART with the propensity score as an additional covariate. These methods are gaining popularity and are widely discussed in the statistics literature. They combine the best of both outcome modeling and treatment modeling.

That said, many researchers prefer to only use matching and other treatment model-focused methods like weighting. I'll provide a short list of some of the primary motivations I've seen:

  • Matching methods can be more robust to model misspecification, making their estimates more trustworthy
  • Matching and weighting involve the assessment and reporting of covariate balance, which provides evidence to the reader that the method has plausibly reduced all the bias due to the measured covariates (this cannot be done with outcome regression)
  • With matching and weighting, one can try many different methods without estimating a treatment effect to find the one that is going to be the most trustworthy. With outcome modeling, you only get one chance, or else you succumb to capitalization on chance and the potential to try many models until the desired effect is found
  • Matching methods are easier to understand and explain to lay audiences
  • Matching and weighting are agnostic to the outcome type, so they can be used in larger models or for outcome types for which g-computation is less straightforward, like survival outcomes
  • Matching and weighting methods are sometimes found to be less biased than g-computation in simulations
  • Matching and weighting are more transparent and customizable; it is easier to incorporate substantive expertise into the way certain variables are prioritized
  • Matching and weighting do not involve extrapolation beyond the region of common support

Hopefully that list gets you started in trying to understand this choice. Unfortunately the question of "should I use matching or g-computation for my data?" is basically equivalent to "what is the correct model for my data?" which is an eternal mystery. The "correct" answer for any given dataset is unknown, and some methods may be better suited for different kinds of datasets based on qualities that are unobservable.

To specifically address your hypotheses:

  1. Yes, sometimes, though combinations of both tend to do best.
  2. Yes-ish; bootstrapping is often used, but not necessarily always valid. For some methods, we can use Bayesian to help. G-computation is not too hard to implement nonparametrically but it often has to be manually programmed.
  3. Same as 2).
  4. Absolutely yes. Just because a method is flexible doesn't mean it will always get the answer right. There is an inherent bias-variance tradeoff that must be managed with all methods. BART tends to do better than other machine learning methods because of how it balances flexibility with precision.
  5. Not really; we know a lot about how to use them, but we know a lot about how they can be improved and using doubly robust methods in many cases dramatically improves their performance.
$\endgroup$
6
  • $\begingroup$ Good answer. As a minor comment, I don't think anything is known theoretically about whether BART has reasonable properties (like nonparametric double-robustness or hitting any semiparametric efficiency bounds) for estimating ATEs. It probably does have some good properties, but there are no theorems. The best existing results for that type of approach is probably obtained by just treating it as a black-box input to DML or targeted learning. $\endgroup$ – guy Oct 30 '20 at 5:44
  • $\begingroup$ Thank you! Yes, I agree with that. I think there are some theorems about its convergence rate, but not specifically applied to effect estimation, and its confidence interval coverage seems to be lacking somewhat. But BART-based g-computation with the propensity score as a covariate is certainly doubly-robust; that's not a property of BART but of g-computation with a propensity score. $\endgroup$ – Noah Oct 30 '20 at 8:39
  • $\begingroup$ Maybe this is just preference, but I take doubly-robust to require, at a minimum, $\sqrt n$ consistency. It’s also not clear what it means for a nonparametric model to be correctly-specified, and when you formalize double robustness as (say) the DML folks have, then BART doesn’t meet that criteria. $\endgroup$ – guy Oct 30 '20 at 14:27
  • $\begingroup$ This was an incredibly thorough answer and I am very thankful. It made it very clear to me what we suspect for theoretical reasons, what we have empirically demonstrated, and what the advantages and disadvantages of each method are. $\endgroup$ – Shade Oct 30 '20 at 18:30
  • $\begingroup$ @Shade glad it was helpful. If you want citations for any of the claims I made let me know (I was too lazy to include them this time). $\endgroup$ – Noah Oct 30 '20 at 19:01
1
$\begingroup$

I found this set of lecture notes to be quite helpful: https://mlhcmit.github.io/slides/lecture15.pdf

There are two common approaches for counterfactual inference, Propensity Scores, and Covariate adjustment.

For Covariate Adjustment you explicitly model the relationship between treatment, confounders, and outcome. Obviously there are lots of options on how to model the relationship, from linear regression, to more advanced techniques, for example random forests and deep learning

To be honest, I'm not sure of why to prefer one approach vs. the other, one thought is perhaps if you're not confident about how to model the causal relationship, or if you've captured all the confounders, but you're able to predict treatment well, then you might favor propsensity score matching?

$\endgroup$
1
  • $\begingroup$ Thanks! The lecture notes helped clarify for me how each one works, and what terminology to use. $\endgroup$ – Shade Oct 30 '20 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.