Appropriateness of propensity score matching when treatment is determinsitic and exogenous (e.g., COVID restrictions)? I recently came across this paper by Sibley et al. in which propensity score matching (PSM) was applied to examine the effects of COVID-19 lockdowns on well-being and government attitudes in New Zealand. Specifically, they treat COVID lockdown as the treatment and use PSM "to match the 1,003 post-lockdown (March 26 to April 12, 2020) respondents with 1,003 from the pool of 23,351 pre-lockdown 'controls'".
Is this a justified use of propensity score matching? I've am perplexed by this specific application. I have not been able to determine exactly why this may (or may not) be an inappropriate application.
The primary issue I see with the approach is that everyone in the 'control' eventually received the treatment, with the treatment being not just "non-random" but entirely deterministic. The best predictor of being in the treatment (i.e., COVID lockdowns) group is simply being in the dataset at all. In other words, because COVID lockdown is exogenous, none of the individual differences captured in the study should be related in any way to the treatment. In turn, I struggle to see how the resulting propensity scores would comprise a valid matching criteria. Is this the only issue?
My assumption is that other matching methods may be far more appropriate here, but I am not entirely sure what these would be or mathematically why. In this example, I would have assumed a fixed-effects regression approach would have likely led to similar conclusions (albeit with likely increased model complexity).
References
Sibley, C. G., Greaves, L. M., Satherley, N., Wilson, M. S., Overall, N. C., Lee, C. H. J., Milojev, P., Bulbulia, J., Osborne, D., Milfont, T. L., Houkamau, C. A., Duck, I. M., Vickers-Jones, R., & Barlow, F. K. (2020). Effects of the COVID-19 pandemic and nationwide lockdown on trust, attitudes toward government, and well-being. The American Psychologist, 75(5), 618–630. https://doi.org/10.1037/amp0000662
 A: The goal of propensity score matching (and any matching) is to create groups that are equated on a list of observed covariates to be matched on. That was their goal in this study, and matching achieved that goal, with very small imbalances remaining between the pre-lockdown and post-lockdown groups. We don't know how large the imbalances were between groups prior to the matching because the authors omitted this from the study, which is inappropriate in my opinion. If the initial imbalances were small, dropping tens of thousands of observations through the matching would have been a huge waste of data. If, for whatever reason, the initial imbalances were large, however, then the propensity score matching was beneficial. Propensity score matching is not necessarily any less appropriate than any other type of matching, since all matching methods serve the same goal. If propensity score matching had been ineffective, the authors might have tried a different method. But there generally are not scenarios where propensity score matching is inappropriate and other forms of matching are appropriate because they all attempt to do the same thing.
I'm not exactly sure what you mean by fixed effects regression. If you mean a regression of the outcome on the treatment with fixed effects for...something (i.e., a difference-in-differences analysis), this is not possible because each participant was only measured once. The groups are independent groups measured at two different times. The authors also considered a within-subjects analysis using a different survey as a supplementary analysis and used paired analyses, which are equivlanet to difference-in-differences. If you just mean a regression of the outcome on the treatment and covariates (i.e., not a multilevel model, i.e., with only fixed and not random effects), propensity score matching and covariate adjustment through regression do the same thing, which is to control for imbalances in the covariates. Regression also increases the precision of the resulting effect estimate and would have been a good idea here, but for the purposes of adjusting for pre-"treatment" differences in the covariates, they are equally valid methods and regression is not more appropriate than matching.
