I have conducted Prospensity Score Matching (in R using the R-package "Matchit"). I used the matching method "nearest neighbor". After matching I compared the treatment and the controlgroup in terms of their outcome variable. For this comparison I used t-test. I discovered that after each matching procedure the results of the t-test changed. To test my assumption that this change in results was due to random selection of the propensity scores (that are used for the nearest neighbor matching) I set the random number generator to a specific seed and conducted the matching procedure several times. By setting the RNG the results didn't differ anymore.
- Confronted with different results after every matching procedure: how do I decide which matching solution I use for further analysis? Is it a valid method to conduct the matching prodecure several times (say 10'000) and report the median of the p- and t-values of the results I get from the several t-tests?