# Bootstrap with replacement with small number of repetitions

In this youtube video about bootstrap resampling, the creator states that when the number of bootstrap processes is small, the distribution of the parameter being estimated can no longer be thought to have a normal distribution (see from the 8:50 mark).

Since the parameter's distribution is not normal, the standard deviation can't be obtained via the standard formula:

The author thus states that one needs to use the CDF to calculate where the 68% confidence interval values are and obtain from there an estimation of the standard deviation.

My question is: where can I get a source for this workaround for obtaining the standard deviation when the number of times the bootstrap process is repeated is small?

I'm interested in this because I'm using bootstrap with replacement to estimate the uncertainty of a parameter and I can't possibly repeat it thousands of times (not even dozens actually) because it would be impossibly time consuming.

• In many case the whole point of using the bootstrap is that the sampling distribution of the statistic is unknown, so in general I would not expect that distribution to be normal (Gaussian) even with a large number of bootstrap replications. – Maarten Buis Feb 5 '14 at 16:28
• @MaartenBuis so are you saying one should always use the CDF method to calculate the uncertainty? – Gabriel Feb 5 '14 at 16:53
• "always" is a big word, but looking at the percentiles (with possible adjustments) for computing confidence intervals fits more naturally within the logic of the bootstrap. – Maarten Buis Feb 5 '14 at 17:17
• from what I heard, isn't the bootstrap generally supposed to be done with replacement though? – user38457 Feb 5 '14 at 19:11

Don't ask us for sources of what somebody claimed on youtube. Ask the creator of the video. The idea of using the normal quantiles to estimate the scale of the distribution obviously comes from robust statistics, and my vague recollection is that it may be due to Tukey. Note that with 20 bootstrap replicates, the 68% confidence interval is going to be far more lousy that the estimate of $s$ that you would get from the standard formula.
The formula for $s^2$ provides an unbiased estimator no matter whether the underlying distribution is normal or not (but it has to have the second moment, of course). What the normality affects is whether that $s^2$ (scaled by $\sigma^2$) will have a $\chi^2$ distribution (and hence the ratios with $s$ in the denominator will have a true $t$-distribution), but unless something really crazy is going on, the $t$-distributions are remarkably robust to non-normality.