2
$\begingroup$

I have some code that does the analysis and creates figures and tables for a publication. It is integrated into my LaTeX build process. Part of the analysis includes a bootstrap comparing the difference in the means of two groups. Apparently, the difference is right around the $p=0.05$ level and every time I run the code I get a different "answer" about what differences are statistically reliable. Based on a couple of runs of the bootstrap, the $p$ values seem to range from about 0.048 to 0.06.

What is the proper way to deal with this? I am thinking I should fix the seed to the random number generator so that it always produces the same result. Do people "trust" that a random seed is actually random and not cherry picked to give a random sample that has statistical reliability? Is the fact that I get different answers depending on the seed and indication that I need more "samples" in my bootstrap?

$\endgroup$
  • $\begingroup$ What orders of magnitude are we talking about? Does it move from $p=.049$ to $p=.052$? Or is it going from $p = .1$ to $p=.01$? $\endgroup$ – Matthew Gunn Oct 4 '16 at 19:39
  • $\begingroup$ @MatthewGunn see edit. $\endgroup$ – StrongBad Oct 4 '16 at 19:54
4
$\begingroup$

Theoretically, more samples in a bootstrap, should lead to a more stable result. Some difference is always to be expected and does not necessarily indicate unreliablity.

As in any scientific report: stuff like this is OK as long as you explain your decisions. If I would write it myself I would fix the seed such that I am certain of the results and mention that with other seeds the result are slightly off. Lastly: the significance level of 0.05 is not some magic line where stuff is suddenly different. In other words: 0.049 is not that different from 0.051 but I think you agree on that.

$\endgroup$
  • $\begingroup$ +1 to @Ivo's remarks. I'm a "belts and braces" kinda guy, so I both set a seed so I get reproducible results, do lots of bootstraps so the results are stable, AND experiment with different seed values to satisfy myself that there isn't something heinous going on based on my seed choices. $\endgroup$ – Charles Eliot Oct 5 '16 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.