# How to deal with multiple “runs” when resampling data

I have some code that does the analysis and creates figures and tables for a publication. It is integrated into my LaTeX build process. Part of the analysis includes a bootstrap comparing the difference in the means of two groups. Apparently, the difference is right around the $p=0.05$ level and every time I run the code I get a different "answer" about what differences are statistically reliable. Based on a couple of runs of the bootstrap, the $p$ values seem to range from about 0.048 to 0.06.

What is the proper way to deal with this? I am thinking I should fix the seed to the random number generator so that it always produces the same result. Do people "trust" that a random seed is actually random and not cherry picked to give a random sample that has statistical reliability? Is the fact that I get different answers depending on the seed and indication that I need more "samples" in my bootstrap?

• What orders of magnitude are we talking about? Does it move from $p=.049$ to $p=.052$? Or is it going from $p = .1$ to $p=.01$? – Matthew Gunn Oct 4 '16 at 19:39
• @MatthewGunn see edit. – StrongBad Oct 4 '16 at 19:54