Propensity score matching with near-perfect differentiation

Question

I am working on an analytics task for my job to match control units to treated units to monitor the effectiveness of a new marketing initiative. We decided to use a propensity score matching method but neither my partner nor I had previous experience with this.

We decided to use the MatchIt package in R which takes in a formula in the form of test ~ x + y + z + ... but we were observing near perfect differentiation, i.e. when you plot the propensity scores, the treated units all have propensity scores very close to 1 and all matched and potential controls have propensity scores very close to 0. Below is an image that shows a jitter plot of the propensity scores for the units.

I cannot find any sources online that have encountered the same problem so I am wondering if it is more appropriate to either

1. use a formula with fewer predictors to get more variation in the propensity scores, or
2. leave the formula as is.

Additionally, if anyone recommends a resource that goes into propensity score matching methods in more depth, we would really appreciate this.

Answer 1

This indicates that the treated and the control units are a-priori not comparable at all. So, solely based on this data on its own, there is no propensity score analysis that can credibly compare them. Simplifying the model may not be appropriate, unless the things you remove are things that you should not have had in the model in the first place (e.g. cost of used treatment, when this is always 0 for control and always >0 for treated).

This is a pretty rare situation, because even when interventions are known to work, there's probably a few people that don't get them (plus a few that probably didn't need them). To use the much used example of parachute use to prevent major trauma or death due to gravitational challenge, obviously the majority of people that get out of a plane without a parachute do this from a plane that has landed and stopped, while those that jump out and deploy a parachute do this from a considerable height. So, you can almost perfectly separate them by height above ground, but you'd still expect a few people that accidentally fell out of planes at great heights to be in a sufficiently comprehensive dataset. Plus, there's probably people that tried out parachutes from really low heights (e.g. those in this study).

Other options could include to make more modeling assumptions. E.g. if you think you have a correct data generating model (e.g. a model that really well captures the results of a physical simulation model for how people jump/fall out of air planes and get injured or not depending on a representative sample of possible landing areas) and can then say something about what would have happened in situations that you never observed. Obviously, that this model really correctly describes the situation then becomes a key assumption.

There are a number of good books in the epidemiology literature, including some that are legally available for free such as The Effect: An Introduction to Research Design and Causality or Causal Inference: What if.

Answer 2

There are a few things that might be going on here.

First is that you don't have overlap in your sample, meaning the control units are so fundamentally different from the treated units that you cannot make a valid causal inference without extrapolating heavily. In this case, you're stuck, and you should either give up on using this data or use a method that can extrapolate, such as regression, acknowledging that the extrapolation will yield inaccurate standard errors that do not account for this extrapolation and the results will be highly model-dependent. In technical terms, this is known as a "positivity violation". Positivity is a critical assumption in causal inference that assumes each unit has a positive probability of being either treated or untreated. See more details on positivity here. To assess whether you have a positivity violation, you should assess balance in your sample prior to matching using cobalt. Using cobalt::bal.plot() on each covariate can reveal if the distributions of each covariate differ fundamentally between the treatment groups. If they do, you have a positivity violation.

Second is that you overfit your propensity score model. This means there may be overlap in your groups, but the propensity score model you used assigned propensity scores of 0 to the control group and 1 to the treated group. There are a few ways to address this. One is to use a different propensity score model. For example, bias-reduced logistic regression (e.g., using the brglm2 package) can yield more accurate propensity scores that avoid overfitting. Another option is to use a matching method that doesn't rely on propensity scores. This could involve either using a distance measure other than the propensity score distance (e.g., the Mahalanobis distance) or using a matching method that doesn't rely on the propensity score at all, like cardinality matching. Both of these are available in MatchIt. It is almost never the case that the default method of 1:1 propensity score matching using a logistic regression propensity score is optimal.

There is so much free literature on propensity score matching on the MatchIt website, which includes extensive bibliographies on using the method.