I have some code that does the analysis and creates figures and tables for a publication. It is integrated into my LaTeX build process. Part of the analysis includes a bootstrap comparing the difference in the means of two groups. Apparently, the difference is right around the $p=0.05$ level and every time I run the code I get a different "answer" about what differences are statistically reliable. Based on a couple of runs of the bootstrap, the $p$ values seem to range from about 0.048 to 0.06.
What is the proper way to deal with this? I am thinking I should fix the seed to the random number generator so that it always produces the same result. Do people "trust" that a random seed is actually random and not cherry picked to give a random sample that has statistical reliability? Is the fact that I get different answers depending on the seed and indication that I need more "samples" in my bootstrap?