Bootstrapped Propensity Score Matching Assume we have a sample where the treatment is present for only a small fraction of the sample, and we want to exploit the treatment using propensity score matching. However, since the treatment group is small, matching with a caliper distance will result in a completely different matched sample everytime and, thus, the randomness is results is huge?
After reading some literature, I found mixed arguments, but: is it common practice to, for example, repeat the caliper matching 1,000 times and estimate average coefficients for the matched sample (without bootstrapping the original sample)? And, subsequently, estimate the standard error as the standard deviation of all average coefficients? Argumentation would of course be to get unbiased estimates, since the matching is always random (in caliper).
I found some articles that state that similar methods yield accurate estimates of the population parameters, however, I don't find many practical evidence of reseachers doing this method, e.g.:


*

*Austin, P. C., & Small, D. S. (2014). The use of bootstrapping when using propensity-score matching without replacement: a simulation study. Statistics In Medicine, 33(24), 4306-4319

*Bai, H. (2013). A Bootstrap Procedure of Propensity Score Estimation. Journal Of Experimental Education, 81(2), 157-177.


Thanks ;)
 A: That is a lot of trouble.  I fear any method that is arbitrary and results in discarding observations that are not incomparable with regard to propensity.  There are many reasons to do either straight covariate adjustment, or covariate adjustment using the logit of the propensity score.  These are detailed in BBR Section 10.1 and Chapter 17.
A: The bootstrapped distribution of effect estimates using your approach cannot be interpreted as an approximation to the population sampling distribution of effects from either a sampling or randomization perspective, so it would be inappropriate to use the standard errors you got from that method as standard errors of the effect estimate.
The approach to average the effect estimates seems reasonable to me; it's a version of multiple imputation, where you are imputing the missing potential outcomes multiple times with different matched units in each "imputation" (.e., each bootstrap iteration). It would then be appropriate to apply Rubin's rules to get an effect estimate and standard errors from the imputed samples. Your standard error would include within-imputation variability due to sampling/randomization AND across-imputation variability due to the variation in matched units. The variability in your bootstrap distribution of effect sizes would be the main factor in this second component, but you'd still need to compute the first component using traditional methods (e.g., Abadie and Imbens SEs, etc.). This approach has not been studied, but this might make an interesting research project. Don't take my word for it; this is intuition based on my understanding of multiple imputation and related processes of arranging data within your sample (e.g., parceling in confirmatory factor analysis).
I think a better approach would be to use a deterministic matching algorithm and using all matched units. For example, if 5 control units fall within the caliper of 1 treated unit, use all 5 control units as matches. This is called variable matching and has been shown to be more effective than 1:1 matching at reducing bias and variability.
