I typically deal with data where multiple individuals are each measured multiple times in each of 2 or more conditions. I have recently been playing with mixed effects modelling to evaluate evidence for differences between conditions, modelling individual as a random effect. To visualize uncertainty regarding the predictions from such modelling, I have been using bootstrapping, where on each iteration of the bootstrap both individuals and observations-within-conditions-within-individuals are sampled with replacement and a new mixed effect model is computed from which predictions are obtained. This works fine for data that assumes gaussian error, but when the data are binomial, the bootstrapping can take a very long time because each iteration must compute a relatively compute-intensive binomial mixed effects model.

A thought I had was that I could possibly use the residuals from the original model then use these residuals instead of the raw data in the bootstrapping, which would permit me to compute a gaussian mixed effect model on each iteration of the bootstrap. Adding the original predictions from the binomial model of the raw data to the bootstrapped predictions from residuals yields a 95% CI for the original predictions.

However, I recently coded a simple evaluation of this approach, modelling no difference between two conditions and computing the proportion of times a 95% confidence interval failed to include zero, and I found that the above residuals-based bootstrapping procedure yields rather strongly anti-conservative intervals (they exclude zero more than 5% of the time). Furthermore, I then coded (same link as previous) a similar evaluation of this approach as applied to data that is originally gaussian, and it obtained similarly (though not as extreme) anti-conservative CIs. Any idea why this might be?

  • $\begingroup$ hm, I just noticed that in the data generating code for both cases I didn't actually add any between-individual variability that one is typically interested in eliminating by modelling individuals as random effects. I'll see if adding this variability changes the outcome; back in a few hours... $\endgroup$ Commented Jul 4, 2011 at 3:25
  • $\begingroup$ If I remember correctly, bootstrap makes the estimate closer to the true population estimate. It doesn't say anything about the confidence interval. (cf. Kesar Singh, On asymptotic accuracy of Efron's bootstrap. Ann. Statist., 1981, 9, 1187-1195) $\endgroup$
    – suncoolsu
    Commented Jul 4, 2011 at 8:54
  • $\begingroup$ @me: I can confirm that the addition of between-individuals variability in the data generation function does not improve performance of the bootstrap. I've uploaded the code I used to confirm this to the gist linked in the original post. $\endgroup$ Commented Jul 4, 2011 at 10:57
  • $\begingroup$ @suncoolsu: I'm pretty sure that bootstrapped confidence intervals have been standard for quite some time. Efron mentions them in his 1978 paper describing the bootstrapping procedure generally, then he had a bunch of papers in the 80s and 90s on tweaks of the bootstrap procedure for more accurate confidence intervals (bias correction, accelleration, studentizing, etc). $\endgroup$ Commented Jul 4, 2011 at 11:27
  • 1
    $\begingroup$ Have you read the following paper by Morris: "The BLUPs are not the best when it comes to bootstrapping". It may relate to your work. link $\endgroup$
    – julieth
    Commented Aug 14, 2012 at 20:39

1 Answer 1


Remember all bootstrap confidence intervals are only asymptotically at the stated confidence level. There are also a slew of possible methods for selecting bootstrap confidence intervals Efron's percentile method, Hall's percentile method, double bootstrap, bootstrap t, tilted bootstrap, BC, BCa and maybe a few more. You haven't told us which method you use. Schenker's paper in JASA 1985 showed that for certain chi square distributions the BC bootstrap confidence interval undercovered the advertised percentage. In small sample size problems this problem can be severe. LaBudde and I have two papers showing how in small samples even BCa can have very poor coverage when estimating a variance from a lognormal distribution and a similar problem exists for testing equality of two variances. This is just for a simple problem. I expect the same thing can happen with residuals from mixed models. In our new book "An Introduction to Bootstrap Methods with Applications to R" published by Wiley in 2011 we cover this topic in Section 3.7 and provide references. The surprise is that the percentile method sometimes does better than the higher order accurate BCa method, when the sample size is small.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.