Strange pattern in standard deviation confidence interval estimation via bootstrapping

I wanted to estimate confidence interval for standard deviation for some data. R code looks like follows:

library(boot)
sd_boot <- function (x, ind) {
res <- sd(x$ReadyChange[ind], na.rm = TRUE) return(res) } data_boot <- boot::boot(data, statistic = sd_boot, R = 10000) plot(data_boot)  And I've got the next plot: I'm stuck with interpreting this histogram of bootstraps correctly. Every other set of similar data shows normal distributions of bootstrap estimates... But not this. By the way, this is actual raw data: > data$ReadyChange
[1] 27.800000  8.985046 11.728021  8.830856  5.738600 12.028310  7.771528  9.208924 11.778611  6.024259  5.969931  6.063484  4.915764
[14] 12.027639  9.111146 13.898171 12.921377  6.916667 10.764479  6.875000 12.875000  7.017917  9.750000  7.921782 12.911551  6.000000


Can you please help me with interpretation of this bootstrap pattern?

• I cannot reproduce your results even copying and pasting the code. I get a very normally distributed histogram. Jan 16, 2017 at 16:29
• @jwimberley , there was a wrong data vector... Thank you for your time for discovering it. Actual data is in post below EDIT. Jan 16, 2017 at 17:01
• pattern confirmed for new data. My guess is that it is caused by the datapoint 27.800000, which is way larger than all the other ones. Jan 16, 2017 at 17:08
• @psarka Confirming that. Removing this point eliminates the odd behavior. The standard deviation of sd without this point is 3.02, but 4.24 with this point. That explains the peaks at 3.02 and 4.24 (point not included in bootstrap; point included in bootstrap). The higher resonances are when this point is included multiple times. Jan 16, 2017 at 17:42
• @mdewey This was based on an observation by psarka which I don't want to take credit for. Jan 16, 2017 at 18:58

2 Answers

You might have a bug in your code, or the bootstrap library does something else than expected.

Edit:

After corrected data was provided, it became apparent that the pattern was caused by one outlier, with each peak corresponding to the different number of times the outlier was selected into a sample.

• Ditto in R: After making the 21 element vector data in the obvious way, create a matrix of indices with inds <- matrix(sample(21,10000*21,replace=TRUE),10000,21) and then lookup the elements of data from each column and find the standard deviation with hist(apply(inds,1,function(ind){sd(data[ind])})). There are not multiple peaks. Jan 16, 2017 at 16:26
• This answer explains and illustrates the problem very well, but provides no guidance or advices about what to do about it in practice. Jan 19, 2017 at 9:32

I am hesitant to put this down as an answer, but to me this seems to be caused by the small amount of datapoints you base your bootstrap on (21, correct me if I'm wrong).

To be more precise, to me it seems these specific 21 values, from which you sample, have only a few frequently possible standard deviations (the peaks in your histogram). If the base sample was larger and more diverse, the resulting histogram would be much smoother (and probably more alike the normal distribution you were expecting).

On a general note and assuming I in the right here, this is a good example to show bootstrapping does not solve the problems of having a small sample.

• I like explanations like this, but I can't reproduce the result! Jan 16, 2017 at 16:19
• @NickCox You are absolutely right to point that out. I entered this answer without trying to replicate these results. As can be seen in Psarka's answer (which I have immediately upvoted) there must have been some coding error. Hence, I've learned something as well (to try and replicate such a problem).
– IWS
Jan 16, 2017 at 16:35
• It's key too that bootstrap is necessarily dependent on the original data. Jan 16, 2017 at 16:36
• Of course it is, what point are you making?
– IWS
Jan 16, 2017 at 16:37
• Supporting you! Namely, arguing that artefacts are possible and that vigilance is needed. Jan 16, 2017 at 16:38