# Different results after propensity score matching in R

I have conducted Prospensity Score Matching (in R using the R-package "Matchit"). I used the matching method "nearest neighbor". After matching I compared the treatment and the controlgroup in terms of their outcome variable. For this comparison I used t-test. I discovered that after each matching procedure the results of the t-test changed. To test my assumption that this change in results was due to random selection of the propensity scores (that are used for the nearest neighbor matching) I set the random number generator to a specific seed and conducted the matching procedure several times. By setting the RNG the results didn't differ anymore.

1. Confronted with different results after every matching procedure: how do I decide which matching solution I use for further analysis? Is it a valid method to conduct the matching prodecure several times (say 10'000) and report the median of the p- and t-values of the results I get from the several t-tests?
• I am not sure why this is being voted as off-topic as there does seem to be a statistical question here which is completely independent of which software is being used. – mdewey Feb 9 '17 at 16:34
• It seems that this question is a duplicate to stats.stackexchange.com/questions/118636/… – Viktor Mar 2 at 7:23

## 3 Answers

This happens when you have (at least) two individuals that have the same propensity score. MatchIt randomly selects one to include in the matched set. My recommendation would be to select one matched set and carry out your analysis with it. I agree that trying other conditioning methods such as full matching and IPW would be a good idea. You could report results of various analyses in a sensitivity analysis section.

Edit: This is probably the wrong answer. See Viktor's answer for what is likely the actual cause.

• Thanks Noah for your reply. Your explanation is very helpful. I decided to do a nearest neighbor caliper matching (random order) as suggested by Austin (2014). As you recommended, I selected one matched set and carried out my analysis with it. – Breeze Feb 20 '17 at 9:14
• I think that it is a wrong explanation. Observations with coinciding propensity scores are very-very rear. The thing is that MatchIt randomly selects the order of treated observations for matching. You may fix matching by calling set.seed() before matching. – Viktor Mar 1 at 16:01
• I agree with you @Viktor. I'll edit my answer. – Noah Jun 24 at 22:19

This is a standard behaviour of MatchIt package. It shuffles the observations before matching, i.e., it randomly selects the order of matching for the treated observations. You may use set.seed() function to fix the results. E.g., call set.seed(100) before calling matchit(). Different arguments of set.seed() will correspond to different matchings.

This is a very interesting question. The first explanation I can suggest is that your study is quite small and thus few matching differences are impactful. More in general, nearest neighbor matching is not very accurate. Caliper mathing is more reliable, and possibly the differences you report would decrease or disappear using it (as with using inverse probability treatment weighting). Finally, I am not sure whether you used the t test to compare baseline differences (which is inappropriate, as this should be done computing standardized differences), or for hypothesis testing (in which case a paired test should be used). In any case, the typical reporting approach is simply to report results of a single matching procedure, as long as it is correctly done (eg with caliper matching).

• Thank you! The baseline sample sizes were 1096 (control) and 328 (treatment group). After matching, both group sizes were reduced to 324. I actually did conduct nearest neighbour matching using a caliper of .25 std of the propensity score. I also compared the nearest neigbour matching with and without the caliper- which lead to 4 additional units in each group being discarded. I computed the standardized differences of the means of the covariates before vs. after matching. These values didn't change after each matching but the values in the outcome variable did. – Breeze Feb 10 '17 at 9:39
• @Breeze I see. Have you tried 1:2 matching or IPTW? – Joe_74 Feb 10 '17 at 10:33
• Hi Joe_74, thank you for your reply. I did try exact matching within the nearest neighbour matching. Unfortunately my sample size reduced to 294 units in both groups. If possible, I would like to maintain sample sizes that are above 300. But I haven't come across the inverse probability treatment weighting. Would you recommend it? – Breeze Feb 10 '17 at 10:51
• @Breeze Definitely. IPTW is key to adjust for residual differences in PS. Using it means also you can keep all your cases, not only matched ones. – Joe_74 Feb 10 '17 at 11:29
• this sounds interesting, thanks. I might try it afterwards. If I conduct my matching as I described above (nearest neighbor with caliper), would you advise me to report the results of a single random matching procedure? Since I get different results every time, to pick just the results of one procedure seems just too random for me... what is your opinion on this? – Breeze Feb 10 '17 at 12:40