1
$\begingroup$

I heard some opinions that matching is no good since it excludes some subjects. However, if we use PS adjustment or PS inverse probability weighting, is there a requirement on the degree of overlapping of PS distributions? Would these methods still work if the overlapping is poor?

$\endgroup$
1
$\begingroup$

For propensity score matching, a lack of overlap yields poor balance if matching without replacement, a small sample size and therefore poor precision if matching with replacement, and a small sample size and a change in estimand if using a caliper. For propensity score weighting for the ATT or ATE, a lack of overlap yields poor balance and a small effective sample size. Both methods "work" when overlap is poor, but they may not work well.

I would not say the choice of matching vs. weighting should depend on the degree of overlap, but the methods used to customize your primary adjustment method should. In this answer, I go into detail on several ways of customizing your weighting method to handle lack of overlap. These methods include changing the estimand, adjusting the weights once estimated, and estimating the weights without using propensity scores. There are several analogous methods for matching: you can use a caliper to restrict matches to be close to each other (which may improve balance but decrease sample size and change the estimand) and you can use tools other than the propensity score to perform the matching, like cardinality matching, which was specifically designed for matching in the presence of low overlap (but also can change the estimand).

Generally, there is a tradeoff between treatment group balance (the degree to which the covariate distributions in the treatment groups resemble each other), precision (determined by the sample size or effective sample size of the matched/weighted sample), and representativeness (the degree to which the matched/weighted sample resembles the target population). Low overlap causes you to have to manage this tradeoff and sacrifice at least one of these desired qualities. If whatever sacrifice you would make would invalidate your inference (e.g., you must estimate the ATT but the only method that gives you balance and precision sacrifices representativeness), then you have to say that your sample simply does not carry enough information to make a valid inference.

Finally, although propensity score overlap can be a good heuristic to assess the overlap of the covariate distributions, it's very possible to incorrectly estimate the propensity score, making assessments based on it invalid. For example, a highly overfit propensity score, which may predict selection into treatment very well, will likely yield many propensity scores close to 0 and 1, indicating poor overlap, regardless of whether the covariate distributions are overlapping. Rather than think about overlap, I encourage you to think about balance, the potential for balance, and the cost of balance (in terms of precision and retention of representativeness). Usually, when we say we have low overlap, we mean the cost of balance is high. This perspective allows us to consider methods that directly balance covariates and more carefully examine the tradeoffs without the heuristic of the propensity score. Many of these methods have been developed recently and have not gained mainstream exposure.

$\endgroup$
3
  • $\begingroup$ I agree that matching partly depends on the caliper we set. There is perhaps a tradeoff between balance, precision and representativeness. $\endgroup$ – hehe Oct 18 '20 at 18:42
  • $\begingroup$ The following statement is a bit confusing: a highly overfit propensity score, which may predict selection into treatment very well, will likely yield many propensity scores close to 0 and 1, indicating poor overlap, regardless of whether the covariate distributions are overlapping. Since propensity scores depend on covariate distributions, the two should be consistent? If your propensity model is correct and you get one group with propensity scores close to 1 and the other group with propensity scores close to 0, there is a problem with the data: violation of positivity assumption. $\endgroup$ – hehe Oct 18 '20 at 20:33
  • $\begingroup$ Not necessarily. If your propensity score model contains a (near) perfect instrument (i.e., causes selection into treatment but has no effect on the outcome), you will get near-perfect prediction and little overlap in the PS even if there is overlap for all other relevant covariates. Overfitting a PS model, e.g., fitting a model with close to as many parameters as there are participants, will also yield near-perfect prediction even if the covariate distributions overlap well. This is a problem of finite sample size. $\endgroup$ – Noah Oct 18 '20 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.