How to determine an appropriate "closeness" threshold when matching for causal inference?

Question

Say I have a [yes/no] treatment variable (e.g. the customer complained about their order) and I want to estimate the causal impact of this "treatment" on the average customer's future spend. To do so, I match tens of thousands of observations in such a way as to minimize their Mahalanobis distance as calculated across a dozen covariates. To estimate the average treatment effect, I prepare a difference-of-means t-test, but before I implement this test across the "treated" and "control" groups, I need to prune my observations of pairs that are insufficiently similar to serve as an effective control -i.e. I need to make a judgement call on the maximum distance a pair of observations can have before being dropped. It goes without saying, the results of the t-test vary drastically as a function of this threshold.

How do I rigorously determine an appropriate "closeness" threshold in the context of causal inference matching?

Answer 1

There are two qualities on which matched samples should be assessed: covariate balance and remaining (effective) sample size. Covariate balance is the degree to which the covariate distributions are the same between the treatment groups in the matched sample. Remaining sample size is the number of units remaining after discarding unmatched units. Covariate balance is required to eliminate bias due to confounding, and remaining sample size is required to achieve a precise estimate. In many cases, there is a trade-off: discarding units can improve balance but reduces remaining sample size. This is an instance of the fundamental bias-variance trade-off that is ubiquitous in statistics.

Another potentially important feature of the matched dataset is the degree to which it represents the population to which you want your effect to generalize. If you discard units in such a way that the remaining matched sample does not resemble your target population, the estimated effect will not be valid for that population. In general, discarding units moves your sample further from the target population. In some cases, this is not so important because the target population itself may be poorly defined or arbitrary, in which case you can say a treatment effect may exist for some population, but not a specific one. I discuss this a bit in my answer here.

So, the answer to your question is to find the cutoff that ensures balance, retains many units, and ensures the sample resembles the target population. There is no magic number, and the optimal value will vary from dataset to dataset and is in principle unknown to the analyst. A commonly used criterion is to disallow pairs of units that are more than .2 standard deviations of the logit of the propensity score apart from each other. Typically, rather than perform a match and then discard distant pairs, you incorporate this criterion, which is known as a "caliper", into the matching itself; that way you don't discard a unit that might have been a good match for a different unit. Calipers are optional in matching; if your matched sample is well-balanced, there is no need to impose a restriction on the distance between paired units.