Bootstrapping a bootstrap One of the criticisms of using a bootstrap procedure is that the results are not reproducible in the sense that you may come to a different conclusions when you re-run the bootstrap analysis again. 
This happened to me today. I was trying to bootstrap a difference in means and found that I would not reject then null hypothesis using 1000 bootstrap samples. The next time I ran this procedure, I rejected the null hypothesis. So, I got the idea to bootstrap my bootstrap estimates. I found that in 1000 samples, I would reject the null hypothesis 959/1000 times. My interpretation, is that I can be fairly confident that I can reject the null hypothesis. Does this seem like a reasonable approach? If so, is there any literature to support this?
Thanks,
 A: If your sample is such that bootstrap replicates are giving you conflicting answers it means you're Monte Carlo error is too large.  Remember that there is a true bootstrap answer represented by taking all possible resamples.  Since it is impractical to do this usually we do Monte Carlo approximation.  So, its not that  the bootstrap is really giving you a conflict, its just that your decision limit is within your MC error... take more resamples.
A: As others suggested, your should rather increase the number of bootstrap replications. Notice that your idea, to "bootstrap the bootstrap" is in fact increasing the number of replications, since you want to sample, for example, 1000 bootstrap replications 1000 times, so in the end you are going to take the one million samples. Why not just take one million samples? Taking the samples in groups and "all at once" leads to basically the same outcome, but aggregated differently.
Second thing to notice is that the number of bootstrap replications should be planned considering your needs and bootstrap errors, as already noticed by others in comments.
In your case, if you want to to conduct some kind of hypothesis test, say on 1% level of significance, this means that you want to see the outcome considered as your alternative hypothesis in ten out of thousand samples; I'd say that this is not that much. What is you draw twelve such samples? If you want to sample something that extreme, you'd need to draw much more bootstrap replications. One thousand is a nice round number, but it's not big for bootstrap.
