How to adress potential issues with interpretation/generalizability after discarding observations in exact matching?

Question

Observations are discarded if they have no exact match, which potentially leads to a different distribution of the covariates in the matched sample compared to the original sample.

I assume that at some level of dissimilarity (correct me if I'm wrong), this difference would affect the possible interpretations of findings, e.g. an ATE could not be intepreted as the ATE of the original sample anymore. The validity of the findings would be restricted to the matched sample. Which is exspecially prpoblematic if the intention is to generalize from the sample to a population, because now I can't even generalize from the matched sample to the original sample anymore, much less to the population.

Is there a way to tell if the matched and original sample are too different? If so, are there strategies to adress the issue, or should I just switch to a matching method that does not discard (as many) observations?

Answer 1

You have hit on exactly one of the most severe problems with matching. A high profile article in JAMA once excluded 0.9 of the patients from a treatment comparison because of the high ratio of controls to cases, and the authors concluded the treatment did not help. A simple covariate-adjusted analysis of all patients showed a large treatment benefit.

Other then when using n:m matching with varying n and m and allowing matched sets to overlap (which creates the need for a very complex analysis following it) matching is often unscientific and non-reproducible. It is statistically very inefficient, ignores interactions, and ignores outcome heterogeneity. I’ve written about the problems in more detail here and here.

Note that ATE is a problematic metric in many ways unrelated to the current discussion, when there are strong covariate effects on outcome.