Propensity score paradox and propensity score matching

I went over the papers by King and Nielsen (2017) and Ripollone et al (2018) to figure out what is propensity score paradox in propensity score matching. I am confused with how they demonstrate the propensity score paradox with pruning: Quote “For each fully matched data set, matched pairs were ranked in order of decreasing absolute propensity score distance or Mahalanobis distance, and the matched pair with the largest distance was pruned from the data set. Covariate balance was assessed for the remaining data set, then the matched pair with the largest distance in the remaining data set was pruned, and covariate balance was assessed again. This process was repeated until only a single matched pair was left in the data set.”

It sounds like we start with a matched data set. From there, we will start to drop out matched pairs with the largest distance (e.g., remove first 30 observations with the largest distances each time) to observe paradox. Why do not we use different caliper sizes in matching to prune the data? E.g., set caliper size=1/2*standard deviation of PS, 1/4 *std of PS, 1/8, 1/16,…..


1 Answer 1


Removing the farthest pair and second farthest pair, etc., one at a time, is essentially the same as applying a tighter and tighter caliper. Instead of describing their analysis as using a caliper, they consider the number of units dropped, since the specific size of the caliper is arbitrary, and for a given range of calipers, there may be no change in the given sample. For example, if the distance between the members of the farthest pair is .5, and the distance between the members of the next farthest pair is .4, you don't need to check every single caliper between .5 and .4; you can just discard the next farthest pair.

The R package MatchingFrontier allows you to assess the propensity score paradox directly in a matched dataset; see the documentation for makeFrontier.matchit() and the accompanying vignette. The documentation shows an example of plotting the relationships between balance and sample size and between balance and the caliper width, which I copy below.

enter image description here

enter image description here

The plots contain exactly the same information, just with a different scaling of the x-axis. (Note when the caliper is used, the units are in standard deviations of the propensity score, just like in your example.) Both reveal the propensity score paradox, which is active in the dataset used, though only when dropping about 60 pairs or more, or, equivalently, using a caliper less than .1 standard deviations of the propensity score.

Another reason to not focus so strongly on the caliper is that you can perform an analysis analogous to the one in King and Nielsen (2019) even when not matching on a propensity score, which is what they do in the analyses of the real datasets. When matching on the Mahalanobis distance, it doesn't make sense to talk about increase the caliper on the propensity score; that is not the relevant measure and a propensity score might not have been estimated at all. The relationship between balance and remaining sample size, which is the primary characteristic defining the propensity score paradox, can be assessed no matter how the distances between units were computed.

So, to summarize, when a propensity score us used for matching, dropping the farthest pair and then the second farthest pair, etc., is the same as applying a tighter and tighter caliper. It is not described that way because calipers are usually used on the propensity score, but the analysis can be performed no matter what the distance measure is, so it makes more sense to focus the analysis on the number of units remaining.

Note that redoing the matching with a tighter and tighter caliper will only yield the same path as dropping the farthest then second farthest pairs, etc., when matching is done in ascending order of distance, which can be done in MatchIt by setting m.order = "closest".

  • $\begingroup$ I don't find this to be a fully principled approach, and to me any method that drops valid data is not fully scientific. Matching almost almost involves dropping observations just because a previous observation was already successfully matched. This is inefficient and does not always lead to reproducible research. More here. $\endgroup$ Commented Jul 24, 2023 at 11:24
  • $\begingroup$ Somehow, I still struggle to comprehend the rationale behind dropping valuable treated subjects with matched controls falling within a valid caliper size in PS matching. By using this caliper size, we can achieve balance, so what's the point of discarding these matched pairs? It seems like the propensity score paradox is not something one might encounter in practical scenarios unless you push the settings to extremes. $\endgroup$
    – Vincent
    Commented Jul 24, 2023 at 13:00
  • $\begingroup$ Another observation from King and Nielsen's paper, as well as the figures presented, is that imbalance is based on pairwise covariate distance. It may not be fair for ps matching because it doesn't ensure that two matched pairs have similar values of matching factors by design. For instance, an older male could be matched with a younger female as long as they have similar propensity scores. While it's true that covariate matching might be more efficient than propensity score matching, it doesn't necessarily mean that propensity score matching should be dismissed altogether. $\endgroup$
    – Vincent
    Commented Jul 24, 2023 at 13:01
  • $\begingroup$ One more aspect that confuses me is the use of "SATE" in King and Nielsen's paper. Based on their simulation settings, it appears that they are actually using "PATE" since they simulate different datasets. For SATE, the interest is on a particular sample (fixed), and the only random element is the treatment assignment mechanism. I find it perplexing to connect SATE with their simulation settings. It would be great If you could shed more lights to these points $\endgroup$
    – Vincent
    Commented Jul 24, 2023 at 13:04
  • $\begingroup$ It seems like the propensity score paradox is not something one might encounter in practical scenarios unless you push the settings to extremes. I agree, which I discuss in my answer here. The paper doesn't claim to recommend dismissing PSM outright, even though that's how many people interpret it (to be fair, the title is misleading). $\endgroup$
    – Noah
    Commented Jul 24, 2023 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.