6
$\begingroup$

I have developed a recent interest in propensity scores. I have been using the SPSS tool created by Dr. F. Thoemmes to calculate propensity scores using bivariate "treatment" variables (e.g., depression) and several covariates (e.g., age, sex, persons in household). I am then given a resulting propensity score, but am left wondering what to do with it.

I have read that what is typical would be to match two individuals who have nearly identical propensity scores (e.g., two people get group number "17") but who actually differ on your treatment variable (e.g., depression), and then do a paired t-test based on group number and dependent variables (e.g., household income). In this example, we would see how two individuals with all of the same propensities (e.g., age, sex, persons in household) but differing on your treatment variable actually differ on your dependent.

This idea makes sense to me, but the software actually does not do matching based on propensity scores, and I don't know how to match them using SPSS or Excel, and I don't want to currently bother to learn how to do so in another program/language (e.g, R). This laziness, lets call it, has forced me to do more research.

Two authors state: "After the matching is completed, the matched samples may be compared by an unpaired t-test. (“Matching” erroneously suggests that the resulting data should be analyzed as if they were matched pairs. The treated and untreated samples should be regarded as independent, however, because there is no reason to believe that the outcomes of matched individuals are correlated in any way)." (Schafer & hang, 2008). Other research seems to suggest people often input propensity scores in logistic regressions next to their independent variable of interest, and see how the independent variable predicts while "propensity" is controlled for.

Although this line of research is interesting, I must admit I am slightly lost regarding what methods are possible/best in terms of conducting quantitative analyses AFTER propensity score calculations. Any guidance on this matter would be appreciated. I will likely have follow-up questions, too!

EDIT: I want to highlight that I am concerned about what types of inferential analyses to do AFTER I get the propensity scores calculated for each individual. For instance, perhaps I could calculate propensity score of being depressed (yes,no) based on covariates (age, number of people in household, smoking, sex, state). The program calculates a propensity score as a new variable for each individual. AFTER this, I am interested in seeing if depression is associated with household income while "controlling for"/"considering"/"matching" (Chose word based on what method you suggest, perhaps) the effect associated with propensity.

Improve this question
$\endgroup$
2
  • $\begingroup$ There is a script in SPSS used to match the cases based on propensity scores here. $\endgroup$ – Andy W Mar 28 '12 at 1:40
  • $\begingroup$ Thank you! Come to think of it, I think I have seen this before but not applied it. It doesn't seem incredibly easy, but perhaps I could manage after looking at it for a few minutes. $\endgroup$ – Behacad Mar 28 '12 at 2:26
4
$\begingroup$

This is a complicated question. The simple nearest neighbor matching pairs each observation in the treatment group with a single person in control group who has a similar propensity score. Then you compute the difference in outcome $Y$ for each pair, and then calculate the mean difference across pairs. That's your treatment effect. However, it is also possible to match each treated person with multiple untreated folks. Matching using additional nearest neighbors increases the bias, as the next best matches are necessarily worse matches, but decreases the variance, because more information is being used to construct the counterfactual for each treated person. Different matching estimators differ in how they weight the neighbor(s) in calculating this difference.

One important question is whether you can pair the same control group person with more than one treated person, essentially recycling them. Matching without replacement can yield very bad matches if the number of comparison observations comparable to the treated observations is small. It keeps variance low at the cost of potential bias, while matching with replacement keeps bias low at the cost of a larger variance since you are using the same info over and over. That is another trade-off.

But I digress. Here are some ways to do propensity score matching, in increasing order of complexity:

  1. The simplest form of matching is using only one control dude who has the closest propensity score (with or without replacement), and calculating the mean difference for all pairs.
  2. Another strategy is divide the $ps(X)$ into $S$ buckets or intervals. For example, say you have some treated observations with $ps(X)$ between 0.3 and 0.4. Then you take all the control group folks with scores between 0.30 and 0.4 and then use their average $Y$ as the counterfactual. The total treatment effect is $\Sigma_{s}(\bar{Y}_{T=1}-\bar{Y}_{T=0})*w_{s}$, where $w_{s}$ is the the fraction of all treated folks in bucket $s$. For example, you might start with 10 $PS$ buckets and they don't need to have the same width. Note that some treated observations may not have any matches! This is known as the common support problem.
  3. Yet another way would be to grab all control group members within a fixed radius of treated unit $i$ and use them as the counterfactuals. Call them group $J_{i}$. The treatment effect is $\frac{1}{T}\Sigma_{i}(\bar{Y}_{i,T=1}-\bar{Y}_{J})*w_{s}$. The bandwidth problem here takes the form of picking the radius.
  4. Kernel matching. Here you weight the control group observations who are further away in PS less heavily, maybe not at all.

How do you pick a method? All matching estimators are consistent, because as the sample gets arbitrarily large, the units being compared get arbitrarily close to one another in terms of their characteristics. In finite samples, which one you choose can make a difference. If comparison observations are few, single nearest neighbor matching without replacement is a bad idea. If comparison observations are many and are evenly distributed, multiple nearest neighbor matching will make use of the rich comparison group data. If comparison observations are many but unevenly distributed (check the PS kernel densities for the two groups), kernel matching is helpful because it will use the additional data where it exists, but not take bad matches where it does not exist.

One complications is that standard errors don't take into account that you estimated the propensity score (since the real thing is not observed), so they are too small. People either ignore this or bootstrap, which may or may not be bad idea.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you for your reply. You offer some insightful suggestions that I will take into consideration when applying propensity score analyses. I don't want to sound ungrateful, and perhaps I have missed what I am looking for, but do you make suggestions on what to do AFTER propensity scores have been calculated? This is the crux of my dilemma. Perhaps you need to lower yourself to my level of expertise on this matter! $\endgroup$ – Behacad Mar 28 '12 at 1:51
  • $\begingroup$ I think my answer got cut off, so I rewrote it. You need to check the common support condition and do some balancing tests. In practice, trying all methods is usually a good idea to check sensitivity of your conclusions. $\endgroup$ – Dimitriy V. Masterov Mar 28 '12 at 2:17
  • $\begingroup$ Thanks again... I'll admit I am still confused. I use nearest-neighbor matching in the program and I get an output regarding balancing and such. I am then left with a new dataset with only participants who have been matched in the analyses and who are left with propensity scores. AFTER this point when I have the propensity scores for each individual, I am unsure what inferential statistic to conduct (e.g., ANCOVA with propensity score, matched t-test, independent samples t-test, logistic regression, etc.) $\endgroup$ – Behacad Mar 28 '12 at 2:25
  • $\begingroup$ Let's say you have 2 treated people in your data, A and B with outcomes $Y_{A}=20$ and $Y_{B}=10$ and propensity scores $P_{A}=.5$ and $P_{B}=.2$. You also have 2 control group individuals, say C and D, with $Y_{C}=15$ and $Y_{D}=7$ and propensity scores $P_{A}=.35$ and $P_{B}=.1$. Then I say that A is closer to C and B is more like D in terms of propensity score. So the average treatment effect would be $(1/2)*(20-15+10-7)=4$. Does that make sense? $\endgroup$ – Dimitriy V. Masterov Mar 28 '12 at 2:42
  • $\begingroup$ To get the standard errors to see if the TE is significant, people will usually bootstrap, though I think there may be an analytic solution as well. $\endgroup$ – Dimitriy V. Masterov Mar 28 '12 at 2:58
1
$\begingroup$

You may want to consider other strategies based on propensity scores, like including them as model covariates, or very similar concepts, like Inverse-Probability-of-Treatment weights. These might work in situations where you can't, or don't want, to deal with matching.

This seems like a decent overview.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. This sounds like a very plausible and practical application. I would need to look into this possibility more to understand the rational behind its use and when it is appropriate, but it is a place to start. $\endgroup$ – Behacad Mar 28 '12 at 16:47
0
$\begingroup$

It is not recommended to include PS as a covariate in an outcome model. You might want to consider a stratified analysis based on strata of the PS.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.