8
$\begingroup$

Suppose I use propensity score matching to create a dataset of treatment and control observations. Then I run OLS regression with some covariates that were not necessarily included in the propensity score model. Do I need to adjust the standard errors in some fashion?

$\endgroup$
1
  • 3
    $\begingroup$ The short answer is yes, because the (a) PS is estimated and (b) you are doing matching. The remedy depends on the details of the procedure. How many matches, with or without replacement, what your OLS looks like, etc.. It would be helpful if you provide those details. $\endgroup$
    – dimitriy
    Commented Oct 20, 2020 at 4:28

1 Answer 1

14
$\begingroup$

Following up on Dimitriy's comment, which I agree with. There are (at least) three sources of uncertainty when performing a propensity score matching analysis: 1) the estimation of the PS, 2) the matching, and 3) sampling variability. I have been writing a review of uncertainty estimation after matching so I'll briefly share those findings here.

The way standard errors must be estimated depends on how the matching was performed. For many forms of matching, we only have simulation evidence of how to proceed; for others, we have analytic expressions; and for others, we have both. After k:1 matching without replacement (including 1:1 matching), the evidence points to using a cluster-robust standard error with pair membership as the clustering variable. Austin and Small (2014) and many of Austin's other papers confirm this using simulation evidence, and Abadie and Spiess (2020) derive this analytically. Both papers also point to the block bootstrap as another solution, in which pairs are sampled with replacement from the matched dataset, and effects are estimated within each bootstrap sample to form the sampling distribution. This is statistically equivalent to a cluster-robust standard error.

There is some debate about how to account for the estimation of the propensity score. Abadie and Imbens (2016) proved analytically that when matching with replacement, ignoring the propensity score estimation makes inferences conservative when estimating the ATE but could go either way when estimating the ATT. However, when Bodory et al. (2020) performed a simulation study attempting to examine the performance of Abadie and Imebns' proposed standard error estimator that accounts for propensity score estimation, they found it to be anti-conservative, and that methods that ignored propensity score estimation performed better empirically. Austin's simulations also indicate that ignoring the estimation of the propensity scores and using cluster-robust standard errors tends to be sufficient.

Finally, it should be known that the standard error depends less on the method used to estimate it when covariates are included in the outcome model. Abadie and Spiess (2020) derived this for matching without replacement, and Hill and Reiter (2006) demonstrated this with simulations for matching with replacement.

What should you do? Include covariates in your outcome model, especially covariates with remaining imbalance or that are highly predictive of the outcome, and use a cluster-robust standard error estimator to estimate the standard error. You can cite Abadie and Spiess (2019) and Austin and Small (2014) to justify this choice.


I'll show you how to implement this in R. Use match.data() on the matchit object to extract the matched dataset. Then use the following code to estimate the effect and its standard error (letting Y be the outcome, A the treatment, X1 and X2 the covariates, and m.data the output of match.data()):

fit <- lm(Y ~ A + X1 + X2, data = m.data, weights = weights)
lmtest::coeftest(fit, vcov. = sandwich::vcovCL, cluster = ~subclass)

Remember only to interpret the coefficient on treatment and not those on the covariates.


Austin, P. C., & Small, D. S. (2014). The use of bootstrapping when using propensity-score matching without replacement: A simulation study. Statistics in Medicine, 33(24), 4306–4319. https://doi.org/10.1002/sim.6276

Abadie, A., & Imbens, G. W. (2016). Matching on the Estimated Propensity Score. Econometrica, 84(2), 781–807. https://doi.org/10.3982/ECTA11293

Abadie, A., & Spiess, J. (2020). Robust Post-Matching Inference. Journal of the American Statistical Association, 0(ja), 1–37. https://doi.org/10.1080/01621459.2020.1840383

Bodory, H., Camponovo, L., Huber, M., & Lechner, M. (2020). The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators. Journal of Business & Economic Statistics, 38(1), 183–200. https://doi.org/10.1080/07350015.2018.1476247

Hill, J., & Reiter, J. P. (2006). Interval estimation for treatment effects using propensity score matching. Statistics in Medicine, 25(13), 2230–2256. https://doi.org/10.1002/sim.2277

$\endgroup$
8
  • 1
    $\begingroup$ Great response. Does your recommendation hold when matching with replacement (1:1 or k:1)? If so, how would you define the clusters in that case? $\endgroup$
    – Pusto
    Commented Oct 21, 2020 at 0:18
  • 1
    $\begingroup$ Also makes me think, why match at all when you could just IPW, in which case it’s pretty clear how to account for uncertainty in pscores via the influence function. $\endgroup$
    – Pusto
    Commented Oct 21, 2020 at 0:21
  • 2
    $\begingroup$ I agree, though matching has some robustness properties that weighting lacks, and robust versions of weighting like entropy balancing and machine learning-estimated weights don't have well described or accessible variance estimators either. Thanks largely to Peter Austin, we actually have a lot of simulation evidence of the performance of variance estimators for matching in finite samples. $\endgroup$
    – Noah
    Commented Oct 21, 2020 at 7:22
  • 1
    $\begingroup$ There have been some studies on this. Austin & Small (2014) examine this for matching w/o replacement for the ATT and find it is conservative. Abadie & Imbens (2008) prove analytically that it is invalid for matching w/ replacement for the ATT, but Hill & Reiter (2006) find it performs well in simulations. It is computationally way more intensive (have to estimate PS and do matching in every replication), whereas the cluster bootstrap performs well empirically and is very quick to implement. So I would just use it if matching w/o replacement, and standard bootstrap if matching w/ replacement. $\endgroup$
    – Noah
    Commented Apr 11, 2023 at 4:37
  • 1
    $\begingroup$ @RobertF Almost all the references in this answer have something to say about the standard bootstrap, so you should read them. Most are non-technical. $\endgroup$
    – Noah
    Commented Apr 11, 2023 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.