# Diagnostics for Bootstrap

I would like to know if there are any agreed on diagnostics one should run after any simple bootstrapping procedure. For example, I bootstrap a quite simple parameter like a median or a correlation to get a standard error for these. The only thing I found so far is to check the distribution of the bootstrapped results to see whether they are normally distributed. Then I found some complex papers that were absolutely unintelligible to me and seemed to cover expert topics. I also found this paper here, which I still need to read: http://www.timhesterberg.net/bootstrap#TOC-What-Teachers-Should-Know-about-the-Bootstrap:-Resampling-in-the-Undergraduate-Statistics-Curriculum

Many novice statisticians mistakenly think you get a bootstrap sample and calculate a mean and SD to do significance testing and confidence limits. This is called the normal bootstrap. The only reason for its persistence is conceptual ease and computational efficiency.

In general the normal bootstrap shouldn't be used. The ways to calculate confidence intervals and significance tests with the bootstrap are: studentized, percentile, bias corrected accelerated (BCA), and double bootstrap. See a relevant post here: Computing p-value using bootstrap with R

If you use a correct bootstrap procedure, there's still no free lunch: some statistics still have invalid tests and summaries. At that point, the only way to know if the bootstrap method actually works is by performing a set of simulation studies. There's no way to inspect, without simulating data, whether a statistic is in fact pivotal, or if the test is of adequate power. This may have been why John Tukey famously declared "[The Bootstrap] should be called the shotgun because it blows the head off of any problem you have, provided you're willing to put the pieces back together."

• Thanks for that post! I am generally aware of these advanced bootstrap methods but wanted to start simple to understand the concept. Suppose I use the BCA bootstrap, which is easy in Stata (I am a Stata user). How would I proceed afterwards? Probably there is no implementation in Stata directly, but what would I need to simulate then? – unistata Mar 21 at 20:00
• @unistata that's probably too broad a question to be helpful on this site. Use the help manual, it's well written and the examples are plentiful stata.com/manuals13/rbootstrap.pdf – AdamO Mar 25 at 15:44

When we take many re-samples for a nonparametric bootstrap, it is important to be mindful of how many uniquely different values of the bootstrapped quantity are actually generated. So I consider counting them to be an important diagnostic.

As an almost-trivial example, I illustrate with data x, which are an sample of size $$n = 50$$ from an exponential population with $$\mu=10$$ (expressed to 8 places). When I took $$B = 10,000$$ re-samples with replacement, I got $$10,000$$ uniquely different sample means, but only 136 uniquely different sample medians.

Details of the simulations in R are shown below:

# re-sampling medians
set.seed(1234)
x = rexp(50, .1);  h.obs = median(x)
eta = qexp(.5, .1); eta
[1] 6.931472
h.re = replicate( 10^4, median(sample(x, rep=T)) )
length(unique(h.re))
[1] 136

# re-sampling means
set.seed(1234)
a.obs = mean(x); a.obs = mean(x)
a.re = replicate( 10^4, mean(sample(x, rep=T)) )
length(unique(a.re))
[1] 10000


• What did counting them tell you? That the median is more likely to be bogus/accurate (I could see it argued either way) because of the small number of possibilities observed? – Joseph Garvin Mar 27 at 5:38
• If you take quantiles .025 and .975 of a bootstrap distribution with only 136 values, those quantiles may not provide a trustworthy CI. CIs may differ greatly from one simulation run to the next. Difficulties of bootstrapping with small original samples are exacerbated. – BruceET Mar 27 at 5:50