16
$\begingroup$

Propensity Score Matching at a high level uses a framework of:

  1. Identify potential confounders from the co-variates i.e all factors which can potentially influence the subject being part of experiment group
  2. Calculate Propensity Score = Pr (Subject being part of treatment | co-variates)
  3. Create a model to estimate the membership for each of the subjects of being in treatment.
  4. Subjects get divided into multiple strata of Control / Experiment based on Propensity scores. This would make the groups balanced w.r.t the subjects having similar characteristics in terms of co-variates.
  5. Calculate effect of treatment by averaging the sum of differences in the dependent variable for each strata.

My question is: why is this better than just matching subjects with a treatment with a similar non-treated subject as measured by a distance measure and co-variate values? And then averaging the sum of differences between all pairs of treated-non-treated subjects?

That seems simpler and would appear to address the self-selection bias that PSM is meant to reduce / eliminate.

What am I missing here?

$\endgroup$
1
  • 2
    $\begingroup$ A propensity matched set can include a 20 year old black male and a 50 year old white intersex person so long as they are equally likely to be exposed. $\endgroup$
    – AdamO
    Feb 12, 2021 at 19:33

2 Answers 2

12
$\begingroup$

The procedure you described is not propensity score matching but rather propensity score subclassification. In propensity score matching, pairs of units are selected based on the difference between their propensity scores, and unpaired units are dropped. Both methods are popular ways of using propensity scores to reduce imbalance that causes confounding bias in observational studies.

In propensity score matching, the distance between two units is the difference between their propensity scores, and propensity scores are computed from the covariates, so by propensity score matching, you are matching based on a distance measure and covariate values. There are other distance measures that don't involve the propensity score that are frequently used in matching, like the Mahalanobis distance. Some studies show the Mahalanobis distance works better than the propensity score difference as a distance measure and some studies show it doesn't. The relative performance of each depends on the unique characteristics of the dataset; there is no way to provide a single rule that is always true about which method is better. Both should be tried. You can also include the propensity score as a covariate in the Mahalanobis distance.

If your question is more about why we would ever do propensity score subclassification when we could do propensity score matching, there are a few considerations. As before, you should always use whichever method yields the best balance in your sample. Propensity score subclassification may do a better job at achieving balance in some datasets and propensity score matching in others. There is no reason to unilaterally decide to use one method over another. Subclassification allows you to estimate the ATT or ATE, whereas most matching methods only allow the ATT. Subclassification is closely related to propensity score weighting when used in certain ways, whereas matching typically doesn't assign nonuniform weights to individuals. With matching, you can customize the specification more (e.g., by using a caliper, by changing the ratio of controls to treated, etc.), whereas with subclassification the opportunities for customization are more limited. The distinction between matching and subclassification is blurred in the face of full matching, which is a hybrid between the two that often performs better than each. Some papers have compared the performance of the two methods, but as I mentioned before, it is important not to rely on general results and instead try both methods in your sample.

Check out the documentation for the MatchIt R package which goes into detail on several matching methods and discusses some of their relative merits and methods of customization.

$\endgroup$
1
  • 2
    $\begingroup$ Thanks, this is really helpful! It sounds like yes, we could "skip" the propensity modeling step and just a distance measure, but there is no reason to believe that it would be better for a given sample. I also now appreciate why tutorials I've seen run through multiple methods since a given method might provide better balance and it is not really possible to know beforehand for a given sample. Thanks. $\endgroup$
    – zthomas.nc
    Feb 11, 2021 at 19:03
11
$\begingroup$

Let's step back and think more broadly about how you could match given some data X.

Exact or Cell Matching

This is hard to do with continuous Xs. You could try rounding/discretizing each variable, but that introduces some measurement error. If you choose to proceed anyway, then you interact these new variables to define cells. Here you run into the curse of dimensionality as X gets big. If you have five variables each with three values, you have $3^5= 243$ cells.

So what to do?

Inexact Matching

Inexact matching procedures reduce the dimension of the problem by defining a distance metric on X and then matching using the distance rather than the X. Mahalanobis distance is one common DM. But you can have two observations that are quite far in MD, but have the same probability of treatment. In many applications, if being bald and being chubby both increase propensity to seek treatment, then it could be OK to compare a treated person who is likely to be treated because he is bald with a control person who has a similar probability because he is chubby. In the PSM framework, this creates a larger pool to match from.

Asymptotically, all inexact matching schemes are consistent as they all tend towards exact matches (on X or on the propensity score) as the sample gets larger. However, they can yield very different answers in finite samples and are all biased in finite samples. PSM is maybe less intuitive than finding similar people, but the goal is not to find similar people. It's to find people who have a similar probability of being treated.

$\endgroup$
4
  • 2
    $\begingroup$ +1 Note: Discretizing continuous variables can introduce bias in addition to error. $\endgroup$
    – Alexis
    Feb 11, 2021 at 18:51
  • 1
    $\begingroup$ OK thanks, that is helpful -- if I am understanding you correctly, I see how controlling for "probability of being treated" is a better way to reduce bias from self-selection than controlling for "subject similarity". I also see how some treated users being relatively unsimiliar to all other users makes pure similiarity matching less useful $\endgroup$
    – zthomas.nc
    Feb 11, 2021 at 18:52
  • $\begingroup$ I am not entirely sure if bias is entirely the right way to think about it. Like I wrote, all these things are biased in finite samples. PSM may give you more comparison units, which certainly good for the variance, all else equal. Maybe the additional units get you Asymptopia a bit faster. But you are also changing the estimand when you are excluding fewer unmatched observations, and so I am not sure it's an apples-to-apples comparison anymore. $\endgroup$
    – dimitriy
    Feb 11, 2021 at 19:17
  • $\begingroup$ Yes. A review of the "curse of dimensionality" posts on CV will reveal one practical reason why exact matching is not even possible when there are several variable at different levels (see here: stats.stackexchange.com/questions/15971/…). Even if you only have 5 variables, each at 5 levels (e.g. White, Black, Asian, etc.), you are dealing with $5^5 = 3,125$ possible combinations. If your dataset is only 1,000 observations, it'll likely be hopeless to match everyone from one treatment to the other, especially for rare combinations of the variables. $\endgroup$ Feb 12, 2021 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.