2
$\begingroup$

Say I have a [yes/no] treatment variable (e.g. the customer complained about their order) and I want to estimate the causal impact of this "treatment" on the average customer's future spend. To do so, I match tens of thousands of observations in such a way as to minimize their Mahalanobis distance as calculated across a dozen covariates. To estimate the average treatment effect, I prepare a difference-of-means t-test, but before I implement this test across the "treated" and "control" groups, I need to prune my observations of pairs that are insufficiently similar to serve as an effective control -i.e. I need to make a judgement call on the maximum distance a pair of observations can have before being dropped. It goes without saying, the results of the t-test vary drastically as a function of this threshold.

How do I rigorously determine an appropriate "closeness" threshold in the context of causal inference matching?

$\endgroup$
2
  • 1
    $\begingroup$ Bravo (+1). You just realised that arbitrary thresholds make for poor inferencial procedures. Use a covariate adjustment method and do not prune your data. (I will writing more on this tomorrow) $\endgroup$
    – usεr11852
    Sep 30, 2020 at 22:35
  • $\begingroup$ Thanks! I am not familiar with covariate adjustment methods and would love to learn more. $\endgroup$ Oct 1, 2020 at 16:27

1 Answer 1

1
$\begingroup$

There are two qualities on which matched samples should be assessed: covariate balance and remaining (effective) sample size. Covariate balance is the degree to which the covariate distributions are the same between the treatment groups in the matched sample. Remaining sample size is the number of units remaining after discarding unmatched units. Covariate balance is required to eliminate bias due to confounding, and remaining sample size is required to achieve a precise estimate. In many cases, there is a trade-off: discarding units can improve balance but reduces remaining sample size. This is an instance of the fundamental bias-variance trade-off that is ubiquitous in statistics.

Another potentially important feature of the matched dataset is the degree to which it represents the population to which you want your effect to generalize. If you discard units in such a way that the remaining matched sample does not resemble your target population, the estimated effect will not be valid for that population. In general, discarding units moves your sample further from the target population. In some cases, this is not so important because the target population itself may be poorly defined or arbitrary, in which case you can say a treatment effect may exist for some population, but not a specific one. I discuss this a bit in my answer here.

So, the answer to your question is to find the cutoff that ensures balance, retains many units, and ensures the sample resembles the target population. There is no magic number, and the optimal value will vary from dataset to dataset and is in principle unknown to the analyst. A commonly used criterion is to disallow pairs of units that are more than .2 standard deviations of the logit of the propensity score apart from each other. Typically, rather than perform a match and then discard distant pairs, you incorporate this criterion, which is known as a "caliper", into the matching itself; that way you don't discard a unit that might have been a good match for a different unit. Calipers are optional in matching; if your matched sample is well-balanced, there is no need to impose a restriction on the distance between paired units.

$\endgroup$
3
  • $\begingroup$ Thank you for your response. I was under the impression that propensity score matching was falling out of favor based on work by Gary King. Do you know if this .2 standard deviations criterion is equally applicable in the context of Mahalanobis distance matching? $\endgroup$ Oct 3, 2020 at 17:26
  • 1
    $\begingroup$ I have written about King & Nielsen (2019) here and here. Just because one researcher doesn't like a method doesn't mean it is falling out of favor. That said, I'm not talking about PS matching. You do Mahalanobis distance matching with PS-based calipers. This method is recommended by the inventors of the PS. $\endgroup$
    – Noah
    Oct 3, 2020 at 17:54
  • $\begingroup$ Understood. Thank you for clarifying. $\endgroup$ Oct 3, 2020 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.