I am working with a team of researchers looking to select a control group out of a large population (150,000) to compare with a relatively small treated group (~900). We plan on using a propensity score to match similar treated members to potential control group members, but also exactly match on other factors (i.e. gender). That is, for a group of criteria that are relevant for matching, we have a portion that will be included in the propensity score, and the remainder will be matched exactly. So I have two questions:

  1. Should we create the propensity score (logistic regression) before exactly matching on certain criteria, and use the score as just one matching criterion along with the exact criteria, OR should the scores be created for each group only after exactly matching on certain criteria?

  2. If the propensity score is formed before exact matching takes place, should the exact matching criteria also be in the propensity score? This question arose when members on the team expressed concern that the significance of the exact matching criteria might be higher than the other criteria included in the propensity score. SAS examples of the PSMATCH procedure also show that exactly matched criteria are included in the PS model in their examples (here: https://documentation.sas.com/?docsetId=statug&docsetTarget=statug_psmatch_examples04.htm&docsetVersion=14.3&locale=en)

Any further insight, readings, or suggestions would be helpful. Thanks.

  • $\begingroup$ Do you need to use PSM to begin with? What is the actual problem you try to address through the use of PS? You have 150K controls and 1K treated subjects; are there still issues of common support between covariates? To quote King & Nielsen (2016): "PSM can help the most in data where valid causal inferences are least likely (i.e., with high levels of imbalance) and may do the most damage in data that are well suited to making causal inferences (i.e., with low levels of imbalance)." You seem to be firmly in the later case. Pruning your data discards information. You sure you need to do it? $\endgroup$ – usεr11852 Aug 29 '18 at 22:04
  • $\begingroup$ Matching using a combination of propensity score and exactly-matched criteria is an attempt to, as closely as possible, match treated and control group members to compare pre-treatment/post-treatment values of certain variables. Without any sort of matching, the control group has a much different profile than the treated group on certain variables that are associated with inclusion in the treatment group (poor "covariate balance," as I've heard it termed), so we're trying to improve that balance by selecting a more like control group for comparison from the total population. $\endgroup$ – David Sep 10 '18 at 18:17

Performing the PS estimation separately within each of the groups defined by the exact matching variables is equivalent to estimating a propensity score in the whole sample with the exact matching variables and their interactions with all other covariates included in the PS model and then splitting your data based on the exact matching variables. Remember that the goal of PS matching is covariate balance; use whichever method yields good covariate balance without requiring you to discard too many treated units.

As I mentioned in this post, omitting an important covariate in a propensity score model, regardless of whether you exactly match on that covariate, can yield poorly performing propensity scores that are far from the true propensity scores (because they are estimated with an incorrect, underspecified model). All though with exact matching you'll achieve perfect balance on the exact matching variables, poorly estimated propensity scores may fail to balance your other covariates. But this is an empirical question that depends on your dataset at hand; you should try various ways of estimating the propensity score and matching and pick the one that yields the best balance and remaining sample size.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.