The gist of propensity score matching, as I understand it, is as follows:

You want to estimate the average treatment effect (ATE) of a treatment on some outcome. However, if you simply calculate the difference between the average outcome of the treated and untreated groups, this may be a biased estimate of ATE if factors that influence the outcome variable also influence the probability of receiving the treatment in the first place.

Propensity score matching minimizes this problem by matching treated and untreated observations with similar probabilities of receiving treatment (via logistic regression of treatment status on covariates), and then estimates ATE as the average difference in outcomes among the matched pairs.

So far, so good? This sounds fine conceptually, but where I have trouble is in seeing how the actual mechanics lead to different outcomes for matched as opposed to naive ATE estimation.

To illustrate:

Suppose four individuals, $X_a, X_b, Y_a, Y_b$, where $X$ indicates that the person did not receive the treatment, $Y$ indicates that the person did receive the treatment, the $a$s have similar covariate values to each other, and the $b$s have similar covariate values to each other.

And suppose $F(^*)$ denotes the outcome for which you are attempting to estimate the effect of treatment.

You first estimate ATE naively, looking at the simple difference in the the average outcome of the treated and the average outcome of the untreated.

Naive ATE estimate: $\frac{F(Y_a)+F(Y_b)}2 - \frac{F(X_a)+F(X_b)}2$

Next, you estimate ATE by first matching on propensity score. As mentioned, the subscript indexing each individual reflects covariate values, and so after we run the logistic regression (ignoring sample size issues), we find that $X_a$ and $Y_a$ have similar propensity scores to each other, while $X_b$ and $Y_b$ have similar propensity scores to each other. We proceed to look at the average difference among these matched pairs.

Matched ATE estimate: $\{[F(Y_b)-F(X_b)] + [F(Y_a)-F(X_a)]\}/2$

The problem is that both the naive ATE estimate and the matched ATE estimate are mathematically equivalent!

Now I'm sure I've made a mistake in my formulation of the matched ATE estimate. My question is, where did I go wrong?

P.S: I am aware that propensity score matching can also be used to drop observations that don't have suitable matches, but I want to ignore that case because my understanding of propensity score matching is that it should lead to a different estimate than a naive estimation even if all observations are matched.

  • 5
    $\begingroup$ Short answer is both methods give the same estimate of treatment effect if the sample size in treated and control groups is the same and you match all observations one to one. $\endgroup$ – jsk May 18 '14 at 0:19
  • 1
    $\begingroup$ Thanks @jsk . I must say though that I am surprised by this. While it's nice to know that I did not make a mistake in my example, I'm just surprised that I've never seen this caveat mentioned in any explanation of propensity score matching I've read, as it seems like an important caveat to note. $\endgroup$ – Yakkanomica May 18 '14 at 0:38
  • 4
    $\begingroup$ @Yakkanomica: What would the caveat warn of? No snares here that I can see. And note that the variance of the ATE estimate will differ if you take the pairing into account in the analysis (not everyone does). $\endgroup$ – Scortchi - Reinstate Monica May 18 '14 at 1:02
  • 1
    $\begingroup$ @Scortchi, the caveat would warn precisely jsk's comment: that if you don't drop any observations and match one-to-one, that you will get the same ATE estimate as you would from a naive estimate that makes no attempt to match. Perhaps this is obvious to others, or perhaps it just never comes up in practice, but to me this was certainly a surprise to learn. That is an interesting point about the variance. $\endgroup$ – Yakkanomica May 18 '14 at 1:10
  • 2
    $\begingroup$ Caveats warn of restrictions on the applicability or interpretation of analyses; which is why I say this observation doesn't necessitate any. Perhaps it doesn't come up in explanations because propensity score matching is seen primarily as a way to select a control group & secondarily (if at all) as a way to reduce the variance of ATE estimates by pairing like observations. $\endgroup$ – Scortchi - Reinstate Monica May 18 '14 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.