8
$\begingroup$

I wanted to estimate confidence interval for standard deviation for some data. R code looks like follows:

library(boot)
sd_boot <- function (x, ind) {                
        res <- sd(x$ReadyChange[ind], na.rm = TRUE)
        return(res)            
}
data_boot <- boot::boot(data, statistic = sd_boot, R = 10000)
plot(data_boot)

And I've got the next plot: enter image description here

I'm stuck with interpreting this histogram of bootstraps correctly. Every other set of similar data shows normal distributions of bootstrap estimates... But not this. By the way, this is actual raw data:

> data$ReadyChange
 [1] 27.800000  8.985046 11.728021  8.830856  5.738600 12.028310  7.771528  9.208924 11.778611  6.024259  5.969931  6.063484  4.915764
[14] 12.027639  9.111146 13.898171 12.921377  6.916667 10.764479  6.875000 12.875000  7.017917  9.750000  7.921782 12.911551  6.000000

Can you please help me with interpretation of this bootstrap pattern?

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ I cannot reproduce your results even copying and pasting the code. I get a very normally distributed histogram. $\endgroup$
    – jwimberley
    Jan 16, 2017 at 16:29
  • 1
    $\begingroup$ @jwimberley , there was a wrong data vector... Thank you for your time for discovering it. Actual data is in post below EDIT. $\endgroup$
    – user16
    Jan 16, 2017 at 17:01
  • 2
    $\begingroup$ pattern confirmed for new data. My guess is that it is caused by the datapoint 27.800000, which is way larger than all the other ones. $\endgroup$
    – psarka
    Jan 16, 2017 at 17:08
  • 2
    $\begingroup$ @psarka Confirming that. Removing this point eliminates the odd behavior. The standard deviation of sd without this point is 3.02, but 4.24 with this point. That explains the peaks at 3.02 and 4.24 (point not included in bootstrap; point included in bootstrap). The higher resonances are when this point is included multiple times. $\endgroup$
    – jwimberley
    Jan 16, 2017 at 17:42
  • 1
    $\begingroup$ @mdewey This was based on an observation by psarka which I don't want to take credit for. $\endgroup$
    – jwimberley
    Jan 16, 2017 at 18:58

2 Answers 2

Reset to default
21
$\begingroup$

You might have a bug in your code, or the bootstrap library does something else than expected.

Edit:

After corrected data was provided, it became apparent that the pattern was caused by one outlier, with each peak corresponding to the different number of times the outlier was selected into a sample.

enter image description here

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Ditto in R: After making the 21 element vector data in the obvious way, create a matrix of indices with inds <- matrix(sample(21,10000*21,replace=TRUE),10000,21) and then lookup the elements of data from each column and find the standard deviation with hist(apply(inds,1,function(ind){sd(data[ind])})). There are not multiple peaks. $\endgroup$
    – jwimberley
    Jan 16, 2017 at 16:26
  • 1
    $\begingroup$ This answer explains and illustrates the problem very well, but provides no guidance or advices about what to do about it in practice. $\endgroup$
    – amoeba
    Jan 19, 2017 at 9:32
8
$\begingroup$

I am hesitant to put this down as an answer, but to me this seems to be caused by the small amount of datapoints you base your bootstrap on (21, correct me if I'm wrong).

To be more precise, to me it seems these specific 21 values, from which you sample, have only a few frequently possible standard deviations (the peaks in your histogram). If the base sample was larger and more diverse, the resulting histogram would be much smoother (and probably more alike the normal distribution you were expecting).

On a general note and assuming I in the right here, this is a good example to show bootstrapping does not solve the problems of having a small sample.

Improve this answer
$\endgroup$
8
  • 3
    $\begingroup$ I like explanations like this, but I can't reproduce the result! $\endgroup$
    – Nick Cox
    Jan 16, 2017 at 16:19
  • 1
    $\begingroup$ @NickCox You are absolutely right to point that out. I entered this answer without trying to replicate these results. As can be seen in Psarka's answer (which I have immediately upvoted) there must have been some coding error. Hence, I've learned something as well (to try and replicate such a problem). $\endgroup$
    – IWS
    Jan 16, 2017 at 16:35
  • $\begingroup$ It's key too that bootstrap is necessarily dependent on the original data. $\endgroup$
    – Nick Cox
    Jan 16, 2017 at 16:36
  • $\begingroup$ Of course it is, what point are you making? $\endgroup$
    – IWS
    Jan 16, 2017 at 16:37
  • 1
    $\begingroup$ Supporting you! Namely, arguing that artefacts are possible and that vigilance is needed. $\endgroup$
    – Nick Cox
    Jan 16, 2017 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.