Different results after propensity score matching in R I have conducted Prospensity Score Matching (in R using the R-package "Matchit"). I used the matching method "nearest neighbor". After matching I compared the treatment and the controlgroup in terms of their outcome variable. For this comparison I used t-test. I discovered that after each matching procedure the results of the t-test changed. To test my assumption that this change in results was due to random selection of the propensity scores (that are used for the nearest neighbor matching) I set the random number generator to a specific seed and conducted the matching procedure several times. By setting the RNG the results didn't differ anymore.  


*

*Confronted with different results after every matching procedure: how do I decide which matching solution I use for further analysis? Is it a valid method to conduct the matching prodecure several times (say 10'000) and report the median of the p- and t-values of the results I get from the several t-tests?

 A: This happens when you have (at least) two individuals that have the same propensity score. MatchIt randomly selects one to include in the matched set. My recommendation would be to select one matched set and carry out your analysis with it. I agree that trying other conditioning methods such as full matching and IPW would be a good idea. You could report results of various analyses in a sensitivity analysis section.
Edit: This is probably the wrong answer. See Viktor's answer for what is likely the actual cause.
Edit 2020-12-07: For MatchIt version less than 4.0.0, the only random selection that would occur when nearest neighbor matching was when ties were present or when m.order = "random", which is not the default. If few variables were used in matching, and especially if they were all categorical or took few values, ties are possible. As of version 4.0.0, there are no longer any random processes unless m.order = "random"; all ties are broken deterministically based on the order of the data.
A: This is a standard behaviour of MatchIt package. It shuffles the observations before matching, i.e., it randomly selects the order of matching for the treated observations. You may use set.seed() function to fix the results. E.g., call set.seed(100) before calling matchit(). Different arguments of set.seed() will correspond to different matchings.
A: This is a very interesting question. The first explanation I can suggest is that your study is quite small and thus few matching differences are impactful. More in general, nearest neighbor matching is not very accurate. Caliper mathing is more reliable, and possibly the differences you report would decrease or disappear using it (as with using inverse probability treatment weighting). Finally, I am not sure whether you used the t test to compare baseline differences (which is inappropriate, as this should be done computing standardized differences), or for hypothesis testing (in which case a paired test should be used).
In any case, the typical reporting approach is simply to report results of a single matching procedure, as long as it is correctly done (eg with caliper matching).
