3
$\begingroup$

I am trying to understand how it is justified to use "with replacement" methods in propensity score matching. In the literature there are many statements like this:

However, situations arise when there are not enough controls in the overlapping region to fully provide one match per treated unit. In this case it can help to use some control observations as matches for more than one treated unit. This approach is often called matching with replacement, a term which commonly refers to with one-to-one matching but could generalize to multiple control matches for each control. Such strategies can create better balance, which should yield estimates that are closer to the truth on average. Once such data are incorporated into a regression, however, the multiple matches reduce to single data points, which suggests that matching with replacement has limitations as a general strategy.

  1. How is the bold sentence above proven? I thought we were trying to find the truth, not cook the books that give us what we want.
  2. How, even philosophically, can this possibly be justified? If this is real data, can you go around making clones and claiming the new set means anything? With replacement methods in Monte Carlo world make sense; we are taking limits and we want to preserve uniformity in our sample space - I can't see how that works here.

Any insight would be appreciated.

$\endgroup$
4
$\begingroup$

You're not creating clones in matching with replacement. You're matching one control to several treated units. You don't artificially double the size of your control group if you match each control to two treated units. The method of estimating the standard error for your treatment effect takes the reuse of control units into account.

Imagine the parallel scenario of propensity score weighting. When you up-weight individuals, you don't artificially increase the size of your sample; the standard error for your estimate accommodates the weights so you don't create something out of nothing. In the matching scenario, you can think of it as giving each control unit a weight of 2 (if they are matched to two treated units), but again, the standard error accounts for this weighting and does not allow for your sample size to be artificially inflated.

$\endgroup$
1
  • $\begingroup$ So in my standard error calculation, n, the number of controls is the same. In my 100:1 example, n would be 1 so the standard error would be huge I suppose? I guess that leads to another question: why do you need 1:1 matching? Is the goal to "balance" covariates or find every trial member a buddy? $\endgroup$ – superhero Apr 2 '18 at 16:38
6
$\begingroup$

Matching without replacement can yield very bad matches if the number of comparison observations comparable to the treated observations is small. It keeps variance low at the cost of potential bias.

Matching with replacement keeps bias low at the cost of a larger variance since you are using the same info over and over. So there's no free lunch (as long as you do actually adjust the variance). It is also good practice when matching with replacement to report the distribution of the number of times that untreated units are used in the matching. This is a second guardrail.

Matching without replacement is also order dependent, which doesn't seem like a great feature. Suppose the treated observations have PS = (10, 20, 30) and the untreated units have PS = (5, 14, 27, 28). If we match the treated units in order the selected comparison units are (14, 27 and 28). Now change the order of the treated units to (20, 10, 30). The matched units are then (14, 5, and 28).

Philosophically, imagine doing an experiment where you test two types of fertilizer on two separate plots of land, and you have a single control plot that you leave alone. Is that control plot a reasonable counterfactual for what would happen if you did not apply fertilizer? Seems so, so you can "recycle" it again for each treatment-control comparison. But if you want to say something about fertilizer in general, you would have smaller standard errors if you had a second control plot.

$\endgroup$
9
  • 1
    $\begingroup$ Yeah, that's what I gathered from the literature but if you play that out, I'm still confused. What if you had 100 treated plots of land and 1 control? Of course, you could "recycle" this and use some weighting scheme but in reality there are always masked confounders and any hidden processes are amplified (probably in a nonlinear way). So, what can you really say? How many copies can you use before the sets are meaningless? It all seems so arbitrary. $\endgroup$ – superhero Mar 30 '18 at 15:00
  • 1
    $\begingroup$ You standard errors would be so large you would not be able to say anything definitive. The histogram would be a single bar. It's a trade-off between bias and variance. I can make the same argument that you could be matching some treated observations with PS of 1 to a sequence of untreated units with PS of zero, and that is an apples to orangutans, biased comparison. There is no clear cutoff, and reasonable people can disagree if a particular trade-off is worth it on a particular data and question. Ideally, one should show robustness to different matching methods and show auxiliary evidence. $\endgroup$ – Dimitriy V. Masterov Mar 30 '18 at 15:21
  • $\begingroup$ Also, I agree with you about unobservables and fragility of matching in that case. $\endgroup$ – Dimitriy V. Masterov Mar 30 '18 at 15:22
  • $\begingroup$ could you please explain the part where you say "...cost of a larger variance since you are using the same info over and over"? Is there an intuitive way to understand this? I see it as: If we are using $\tau = \sum_{i=1}^N Y^T_{i}-Y^C_{i}$, then the variance of this will be dependent if more than one $Y^C_{i}$ is the same? $\endgroup$ – user321627 Feb 9 at 9:23
  • $\begingroup$ @user321627 That formula is not how PSM matching generally works. For ATT, the estimator is $$\frac{1}{N_{T}} \sum_{i \in \{D_i=1\}} \left[ Y_{1i} - \sum_{j \in \{D_j=0\}}w(i,j)\cdot Y_{0j} \right],$$ where $w(i,j)$ is the weight placed on the jth observation in constructing the counterfactual for the ith treated observation, $D$ is the treated indicator, and $Y$ is the potential outcome. Matching with replacement means the same control observation can go into the counterfactual for multiple treated observation, so you need to take that into account when estimating the variance. $\endgroup$ – Dimitriy V. Masterov Feb 9 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.