11
$\begingroup$

I have conducted Prospensity Score Matching (in R using the R-package "Matchit"). I used the matching method "nearest neighbor". After matching I compared the treatment and the controlgroup in terms of their outcome variable. For this comparison I used t-test. I discovered that after each matching procedure the results of the t-test changed. To test my assumption that this change in results was due to random selection of the propensity scores (that are used for the nearest neighbor matching) I set the random number generator to a specific seed and conducted the matching procedure several times. By setting the RNG the results didn't differ anymore.

  1. Confronted with different results after every matching procedure: how do I decide which matching solution I use for further analysis? Is it a valid method to conduct the matching prodecure several times (say 10'000) and report the median of the p- and t-values of the results I get from the several t-tests?
Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ I am not sure why this is being voted as off-topic as there does seem to be a statistical question here which is completely independent of which software is being used. $\endgroup$
    – mdewey
    Feb 9, 2017 at 16:34
  • 1
    $\begingroup$ It seems that this question is a duplicate to stats.stackexchange.com/questions/118636/… $\endgroup$
    – Viktor
    Mar 2, 2019 at 7:23

3 Answers 3

Reset to default
3
$\begingroup$

This happens when you have (at least) two individuals that have the same propensity score. MatchIt randomly selects one to include in the matched set. My recommendation would be to select one matched set and carry out your analysis with it. I agree that trying other conditioning methods such as full matching and IPW would be a good idea. You could report results of various analyses in a sensitivity analysis section.

Edit: This is probably the wrong answer. See Viktor's answer for what is likely the actual cause.

Edit 2020-12-07: For MatchIt version less than 4.0.0, the only random selection that would occur when nearest neighbor matching was when ties were present or when m.order = "random", which is not the default. If few variables were used in matching, and especially if they were all categorical or took few values, ties are possible. As of version 4.0.0, there are no longer any random processes unless m.order = "random"; all ties are broken deterministically based on the order of the data.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks Noah for your reply. Your explanation is very helpful. I decided to do a nearest neighbor caliper matching (random order) as suggested by Austin (2014). As you recommended, I selected one matched set and carried out my analysis with it. $\endgroup$
    – Breeze
    Feb 20, 2017 at 9:14
  • $\begingroup$ I think that it is a wrong explanation. Observations with coinciding propensity scores are very-very rear. The thing is that MatchIt randomly selects the order of treated observations for matching. You may fix matching by calling set.seed() before matching. $\endgroup$
    – Viktor
    Mar 1, 2019 at 16:01
  • $\begingroup$ I agree with you @Viktor. I'll edit my answer. $\endgroup$
    – Noah
    Jun 24, 2019 at 22:19
2
$\begingroup$

This is a standard behaviour of MatchIt package. It shuffles the observations before matching, i.e., it randomly selects the order of matching for the treated observations. You may use set.seed() function to fix the results. E.g., call set.seed(100) before calling matchit(). Different arguments of set.seed() will correspond to different matchings.

Improve this answer
$\endgroup$
1
$\begingroup$

This is a very interesting question. The first explanation I can suggest is that your study is quite small and thus few matching differences are impactful. More in general, nearest neighbor matching is not very accurate. Caliper mathing is more reliable, and possibly the differences you report would decrease or disappear using it (as with using inverse probability treatment weighting). Finally, I am not sure whether you used the t test to compare baseline differences (which is inappropriate, as this should be done computing standardized differences), or for hypothesis testing (in which case a paired test should be used). In any case, the typical reporting approach is simply to report results of a single matching procedure, as long as it is correctly done (eg with caliper matching).

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Thank you! The baseline sample sizes were 1096 (control) and 328 (treatment group). After matching, both group sizes were reduced to 324. I actually did conduct nearest neighbour matching using a caliper of .25 std of the propensity score. I also compared the nearest neigbour matching with and without the caliper- which lead to 4 additional units in each group being discarded. I computed the standardized differences of the means of the covariates before vs. after matching. These values didn't change after each matching but the values in the outcome variable did. $\endgroup$
    – Breeze
    Feb 10, 2017 at 9:39
  • $\begingroup$ @Breeze I see. Have you tried 1:2 matching or IPTW? $\endgroup$
    – Giuseppe Biondi-Zoccai
    Feb 10, 2017 at 10:33
  • 1
    $\begingroup$ Hi Joe_74, thank you for your reply. I did try exact matching within the nearest neighbour matching. Unfortunately my sample size reduced to 294 units in both groups. If possible, I would like to maintain sample sizes that are above 300. But I haven't come across the inverse probability treatment weighting. Would you recommend it? $\endgroup$
    – Breeze
    Feb 10, 2017 at 10:51
  • $\begingroup$ @Breeze Definitely. IPTW is key to adjust for residual differences in PS. Using it means also you can keep all your cases, not only matched ones. $\endgroup$
    – Giuseppe Biondi-Zoccai
    Feb 10, 2017 at 11:29
  • 1
    $\begingroup$ this sounds interesting, thanks. I might try it afterwards. If I conduct my matching as I described above (nearest neighbor with caliper), would you advise me to report the results of a single random matching procedure? Since I get different results every time, to pick just the results of one procedure seems just too random for me... what is your opinion on this? $\endgroup$
    – Breeze
    Feb 10, 2017 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.