Strange pattern in standard deviation confidence interval estimation via bootstrapping

I wanted to estimate confidence interval for standard deviation for some data. R code looks like follows:

library(boot)
sd_boot <- function (x, ind) {
res <- sd(x$ReadyChange[ind], na.rm = TRUE) return(res) } data_boot <- boot::boot(data, statistic = sd_boot, R = 10000) plot(data_boot)  And I've got the next plot: I'm stuck with interpreting this histogram of bootstraps correctly. Every other set of similar data shows normal distributions of bootstrap estimates... But not this. By the way, this is actual raw data: > data$ReadyChange
[1] 27.800000  8.985046 11.728021  8.830856  5.738600 12.028310  7.771528  9.208924 11.778611  6.024259  5.969931  6.063484  4.915764
[14] 12.027639  9.111146 13.898171 12.921377  6.916667 10.764479  6.875000 12.875000  7.017917  9.750000  7.921782 12.911551  6.000000


• I cannot reproduce your results even copying and pasting the code. I get a very normally distributed histogram. – jwimberley Jan 16 '17 at 16:29
• @jwimberley , there was a wrong data vector... Thank you for your time for discovering it. Actual data is in post below EDIT. – user16 Jan 16 '17 at 17:01
• pattern confirmed for new data. My guess is that it is caused by the datapoint 27.800000, which is way larger than all the other ones. – psarka Jan 16 '17 at 17:08
• @psarka Confirming that. Removing this point eliminates the odd behavior. The standard deviation of sd without this point is 3.02, but 4.24 with this point. That explains the peaks at 3.02 and 4.24 (point not included in bootstrap; point included in bootstrap). The higher resonances are when this point is included multiple times. – jwimberley Jan 16 '17 at 17:42
• @mdewey This was based on an observation by psarka which I don't want to take credit for. – jwimberley Jan 16 '17 at 18:58

You might have a bug in your code, or the bootstrap library does something else than expected.

Edit:

After corrected data was provided, it became apparent that the pattern was caused by one outlier, with each peak corresponding to the different number of times the outlier was selected into a sample.

• Ditto in R: After making the 21 element vector data in the obvious way, create a matrix of indices with inds <- matrix(sample(21,10000*21,replace=TRUE),10000,21) and then lookup the elements of data from each column and find the standard deviation with hist(apply(inds,1,function(ind){sd(data[ind])})). There are not multiple peaks. – jwimberley Jan 16 '17 at 16:26
• This answer explains and illustrates the problem very well, but provides no guidance or advices about what to do about it in practice. – amoeba Jan 19 '17 at 9:32

I am hesitant to put this down as an answer, but to me this seems to be caused by the small amount of datapoints you base your bootstrap on (21, correct me if I'm wrong).

To be more precise, to me it seems these specific 21 values, from which you sample, have only a few frequently possible standard deviations (the peaks in your histogram). If the base sample was larger and more diverse, the resulting histogram would be much smoother (and probably more alike the normal distribution you were expecting).

On a general note and assuming I in the right here, this is a good example to show bootstrapping does not solve the problems of having a small sample.

• I like explanations like this, but I can't reproduce the result! – Nick Cox Jan 16 '17 at 16:19
• @NickCox You are absolutely right to point that out. I entered this answer without trying to replicate these results. As can be seen in Psarka's answer (which I have immediately upvoted) there must have been some coding error. Hence, I've learned something as well (to try and replicate such a problem). – IWS Jan 16 '17 at 16:35
• It's key too that bootstrap is necessarily dependent on the original data. – Nick Cox Jan 16 '17 at 16:36
• Of course it is, what point are you making? – IWS Jan 16 '17 at 16:37
• Supporting you! Namely, arguing that artefacts are possible and that vigilance is needed. – Nick Cox Jan 16 '17 at 16:38