An example of when bootstrap has less bias than classically approximated estimates?

Question

The recent question "Why does my bootstrap interval have terrible coverage?" has got me wondering if anybody has some really good examples of distributions in which bootstrapping standard errors systematically outperforms classic estimators (I am not sure what the right terminology for the classic set of estimators... perhaps moment approximation estimators?).

In order to respond to the comment that bootstrapped estimators might be less biased by Ben Ogorek, I have also modified the code (originally posted by Flounderer) to include an estimate of bias and mean squared error. After repeating the simulation 10,000 times, it appears to me that the mean squared errors are statistically identical.

Thanks for your consideration in this matter!

tCI.total <- 0
bootCI.total <- 0
m <- 10 # sample size

Trep <- 10000 # number of repetitions of the proceedure
Brep <- 1000 # number of repetitions of the bootrap

sampv <- mbootv <- rep(0,Trep)

true.mean <- exp(2) + 1

# Clear the coverage index values.
tCI.total <- bootCI.total <- 0

for (i in 1:Trep){
  samp <- exp(rnorm(m,0,2)) + 1
  sampv[i] <- mean(samp)

  tCI <- mean(samp) + c(1,-1)*qt(0.025,df=9)*sd(samp)/sqrt(m)

  boot.means <- rep(0,Brep)
  for (j in 1:Brep) boot.means[j] <- mean(sample(samp,m,replace=T))

  mbootv[i] <- mean(boot.means)

  bootCI <- sort(boot.means)[c(0.025*length(boot.means), 0.975*length(boot.means))]

  if (true.mean > min(tCI) & true.mean < max(tCI)) tCI.total <- tCI.total + 1
  if (true.mean > min(bootCI) & true.mean < max(bootCI)) bootCI.total <- bootCI.total + 1 
}
tCI.total/Trep     # estimate of t interval coverage probability
# 0.5634
bootCI.total/Trep  # estimate of bootstrap interval coverage probability
# 0.5416

# Let's look at bias esimate for the sample mean and the bootrapped population mean estimate
(true.mean - mean(mbootv)) # bias estimate of bootstrapped means
# 0.170623
(true.mean - mean(mbootv))^2 + sd(mbootv)^2 # mean squared error of bootstrapped means
# 198.5914

(true.mean - mean(sampv))  # bias estimate of sample means
# 0.170475
(true.mean - mean(sampv))^2 + sd(sampv)^2 # mean squared error of sample means
# 198.4912

Answer 1

Two things that may or may not change the results of your current experiment:

1. [Nitpicky.] I'm pretty sure you should use a runif not rnorm in the exponentiation to generate your sample.

2. I believe I've read that the quantile/percent CI for bootstrapping is not as robust in the face of skewed distributions as we might want to believe.

[EDIT: I've found the reference...]

Check out Chapter 18 (Bootstrap Methods and Permutation Tests) of a book, The Practice of Business Statistics, and the chapter was written by Tim Hesterberg, David S. Moore, Shaun Monaghan, Ashley Clipson, and Rachel Epstein.

As I understand them, they say that the percentile CI is better than some alternatives with skewed data, but there are even better options (BCa, etc) so they recommend that if the normal and percentile methods disagree, use something better than the percentile method. (Which begins to sound a little puzzling to me: is it bootstrapping or a sophisticated CI evaluation that's helpful?)

[EDIT 2] My suggestions don't make much of a difference. However, I did find in the Wikipedia Bootstrap article, it mentions: "When working with small sample sizes (i.e., less than 50), the percentile confidence intervals for (for example) the variance statistic will be too narrow. So that with a sample of 20 points, 90% confidence interval will include the true variance only 78% of the time[27]".