Should you ever use non-bootstrapped propensity scores? I am trying to measure the difference in continuous $y$ given a binary treatment $B$ and I am using the propensity score matching method. As I built the propensity score model I noticed that small changes in some of the variables lead to a really big difference in estimates of the effect of $B$ downstream.
In order to measure this variance I built a bootstrap. Sample $n$ rows with replacement from the data, fit the the propensity score model on the sample, match, and estimate the effect of $B$. Repeat.
This begs the question - is there ever a reason NOT to do this? It seems like this incorporates an extra level of uncertainty that otherwise wouldn't be captured.
 A: It's true that there are several sources of uncertainty in propensity score matching. One is sampling from the superpopulation (which is true of most statistical analyses and is the usual justification for sampling distributions and confidence intervals), but two others are the uncertainty in estimating the propensity score and the uncertainty due to matching. I would not say the solutions to addressing these uncertainties are well understood, but we do have lots of evidence from simulation studies and some analytic derivations to guide us.
Regarding bootstrapping. Bootstrapping the whole process of estimating the propensity score, performing the matching, and estimating the effect is often a good idea. Although Abadie & Imbens (2008) argued analytically that the bootstrap is not valid when performing matching with replacement, simulation studies including Bodory et al. (2020) and Hill & Reiter (2006) have found the bootstrap to be adequate, if not conservative. For matching without replacement, simulations by Austin & Small (2014) examining the performance of bootstrapping have found that the full bootstrap as you described is conservative. From this evidence, we can feel confident in using the bootstrap for propensity score matching.
Austin & Small (2014) and Abadie & Spiess (2019) both observed that a block bootstrap actually approximates the sampling variability of a propensity score matching estimator better than a traditional bootstrap. In the block bootstrap, you perform the propensity score estimation and matching in your original sample and then bootstrap pairs from that sample to estimate the treatment effect. This seems to ignore the uncertainty due to estimating the propensity score, but it turns out this isn't so problematic. Abadie & Imbens (2016) found analytically that treating the propensity score as fixed actually increases the variability of the effect estimate, which was further confirmed by Austin & Small (2014) who compared bootstrapping with the true and estimated propensity score. This same type of relationship has been found with propensity score weighting, which is why failing to account for the estimation of the propensity score actually yields conservative standard error estimates (Lunceford & Davidian, 2004).
So, to answer your question, you can bootstrap the whole process, but you don't have to, and you can validly estimate the variance of the effect estimate by ignoring the variability due to the estimation of the propensity score and instead performing a block bootstrap on the matched pairs. It might be the case that your dataset is weird and the block bootstrap doesn't correctly address the true uncertainty in the effect estimate, in which case it might be beneficial to use an estimation for which uncertainty estimation is well understood, like propensity score weighting.

Abadie, A., & Imbens, G. W. (2008). On the Failure of the Bootstrap for Matching Estimators. Econometrica, 76(6), 1537–1557. JSTOR.
Abadie, A., & Imbens, G. W. (2016). Matching on the Estimated Propensity Score. Econometrica, 84(2), 781–807. https://doi.org/10.3982/ECTA11293
Abadie, A., & Spiess, J. (2019). Robust Post-Matching Inference. 34.
Austin, P. C., & Small, D. S. (2014). The use of bootstrapping when using propensity-score matching without replacement: A simulation study. Statistics in Medicine, 33(24), 4306–4319. https://doi.org/10.1002/sim.6276
Bodory, H., Camponovo, L., Huber, M., & Lechner, M. (2020). The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators. Journal of Business & Economic Statistics, 38(1), 183–200. https://doi.org/10.1080/07350015.2018.1476247
Hill, J., & Reiter, J. P. (2006). Interval estimation for treatment effects using propensity score matching. Statistics in Medicine, 25(13), 2230–2256. https://doi.org/10.1002/sim.2277
Lunceford, J. K., & Davidian, M. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study. Statistics in Medicine, 23(19), 2937–2960.
