# One-to-many matching of propensity scores and average treatment effect on the treated

I've been working on a propensity score matching project, but having no stats classes under my belt I'm struggling to calculate the so called "average treatment effect on the treated" (ATT) so that it will be statistically valid. It seems straight-forward enough when looking at the following formula:

$ATT = E[y1 - y0 | p(x)] = E[y | t = 1,p(x)] - E[y | t = 0, p(x)]$

That is, the average treatment effect on the treated is equal to the expected value ($E$ = average in this case) of the difference between the outcome variable (for example, $y$ = income) for treated and untreated individuals (where, for example, treated individuals ($t=1$) received some sort of training and untreated individuals ($t=0$) did not), conditional on their propensity scores, $p(x)$ (i.e. when their propensity scores match "well enough"). With one-to-one matching of treated and control individuals using nearest neighbour or exact matching, calculating ATT manually in a spreadsheet is fine, but when matching more than one control to each treated individual things seem to get a little tricky. As such, I'm hoping someone may be able to help me figure out what the generally accepted methods for calculating ATT are when dealing with one-to-many matching between treated and control individuals.

• Is your formula measuring the average treatment effect rather than the average treatment effect for the treated? – RobertF Oct 4 '16 at 17:26

All matching estimators for the treatment on the treated effect can be written in the form

$$\frac{1}{n_T} \sum_{i \in \{d_i=1\}} \left[ y_{1i} - \sum_{j \in \{d_j = 0 \}} w_{ij} \cdot y_{0j} \right] ,$$

where $w_{ij}$ is the weight placed on the $j$th untreated observation as a counterfactual for the $i$th treated observation, and $n_T$ is the number of treated persons. The weights satisfy $\sum_j w_{ij}=1$ for all $i$.

Effectively, from each treated observation $i$, you subtract a weighted average of the control observations. Then you take the average of these differences. These weights are specific to observation $i$. Different matching estimators differ in how they construct the weights.

For example, nearest neighbor matching sets the weight to 1 for the single untreated observations closest to $i$ in terms of the propensity score and to 0 for all others. k-NN uses $k$ closest neighbors instead.

Interval matching consists of dividing the range of propensity scores into a fixed number of intervals (which need not be of equal length). An interval-specific estimate is obtained by taking the difference between the mean outcomes of the treated and untreated units in each interval.

Radius/caliper matching takes the mean of the outcomes for untreated units within a fixed radius of each treated unit as the estimated expected counterfactual. You pick the radius.

Kernel matching uses weights that decline with the PS distance. You can think about kernel matching as running a weighted regression for each treated observation using the comparison group data and the regression includes only an intercept term. Here you have to pick the kernel and the bandwidth. Larger bandwidth means further observations will have larger weights.

Local linear matching is very similar, but also included a linear term in PS. Some people will also include higher order polynomial terms.

Finally, you have inverse probability weighting. The basic idea is that you can figure out the expected untreated outcome (in either the treated population or the full population) by reweighting the observed values using the treatment probabilities.

There are some guidelines about how to pick a method here.

There is a list of software and packages that can do matching here. Stata also now has native PSM estimators. In my experience, replicating the output by hand is often very hard once you go past the simplest estimators. However, you can also find examples with output for all of these online, so even if you don't have the software, they will give you a useful benchmark since you can usually track down the data.

• Having completed a propensity score analysis, with good match balance between treated and control units, is there anything further that is needed in order to say whether the results are statistically significant? Or is it enough to have good covariate balance? – InquisitiveInquirer Jul 20 '16 at 9:49
• @user7388 You will need standard errors. You will want to make sure the distributions of PS for T and C have substantial overlap. Finally, balancing does not help with statistical significance (see section 3.4 in the Caliendo & Sabine 2008 JES paper. Finally, I also like to show results from false palcebo tests (repeat the analysis before the effect takes place and ATT=0). – Dimitriy V. Masterov Jul 20 '16 at 18:08
• If you estimate an effect where you know there should be none, PSM will not work because you have selection into treatment on unobservables variables that PSM cannot help with. – Dimitriy V. Masterov Jul 20 '16 at 18:09
• When you say that I will need standard errors, do you think you could explain how I would implement that a little further, as having no stats background makes such things rather confusing :) Also, when you mention that the PS distributions for T and C have substantial overlap, is this the same as having good covariate balance or is it something more nuanced? – InquisitiveInquirer Jul 21 '16 at 7:34
• @user7388 Standard error will be reported by any of the stats packages linked above. Don't use Excel. If it is large relative to your estimate of ATET, you cannot rule out that the effect is zero with the data you have. Overlap means that for treated observations with PS near some value x you have some untreated observations with PS near x, so you can make comparisons that is apples to apples. Balancing is about getting the model of propensity for treatment correct. Honestly, PSM is pretty advanced stuff to be doing without statistical training. You're jumping in the deep end of the pool. – Dimitriy V. Masterov Jul 21 '16 at 8:03

Edit - Please find below comprehensive list of options - http://www.biostat.jhsph.edu/~estuart/propensityscoresoftware.html