# Bootstrap confidence intervals interpretation; too large when testing sample mean

I implemented basic, Studentized and percentile bootstrap methods (in Matlab), to do hypothesis testing that a sample mean is significantly different than zero. (I realize from the limited research I've done so far that this may not be the best approach, but I'm currently only trying to replicate previous work.) However, my question is about the interpretation of bootstrap confidence intervals (CI), and not obtaining the expected result when testing my function with random data.

My understanding of the CI interpretation is that if one repeats many time sampling from the population and creating a CI, then the true population parameter will be in the CI "alpha-proportion" of the time. But when I do my test with either uniform or normally distributed random data centered at 0, with alpha 0.05, I get only a proportion of cases of about 0.006 with zero outside the CI. With alpha 0.25, I only find it 0.11 of times outside. The same is true with all 3 bootstrap methods mentioned above. It seems that the CIs I obtain by bootstrap are therefore much too wide. I do a two tailed test at level alpha (such that alpha/2 is on each side), and I checked that both sides contribute.

Is there an explanation for this? Or should I continue trying to find a bug in my code?

Turns out the "wrong" false positive proportions I was seeing were indeed indicative of errors in my code. Thus this was more a programming than a statistical interpretation issue. But I guess this particular error illustrates how sensitive the bootstrap method can be. My resampling method is apparently not quite equivalent to uniform random sampling with replacement, and it generates a wider bootstrap distribution.

Also, make sure you are resampling the proper way. It sounds like you are resampling from the population for each iteration of your bootstrap procedure, when the resampling should be done from the sample you initially took from the population for each iteration.

Here is some pseudocode for how the procedure should work:

1. Take an initial sample from the overarching population with specified distribution (i.e. normal, uniform) of size n. I'll call this the Original Sample.

2. Take a sample, with replacement, also of size n, from the Original Sample. I'll call this the Bootstrap Sample.

3. Compute the mean for the Bootstrap Sample.

4. Repeat steps 2 and 3 a number of times, for simplicity I will use n here as well, but the size here seems flexible from my understanding. This results in n bootstrap means.

5. Order the bootstrap means from least to greatest in order to determine the percentile(s) you are interested in. From here you can determine your alpha% Confidence Interval.

If you do this, your observed probabilities should start to line up with what the confidence intervals tell you. For example if you do these steps 100 times and obtain 100 different 95% confidence intervals using the percentile method, about 5 of the 100 will not contain 0. In general, if you repeat the steps N times, about alpha% of your (1-alpha)% CIs will not contain 0.

• Sorry if my explanation was confusing, I did the bootstrap correctly as you explain. But to test it, I had to repeat it many times (thus sampling from the population and getting a CI each time as I said). It was the specific way in which I pick elements to build the bootstrap samples (your step 2) that was slightly wrong: each distinct bootstrap sample possibility had the same probability, thus wrongly emphasizing having duplicate elements compared to proper random resampling with replacement, thus widening the distribution and CIs. Jan 29, 2015 at 23:51