# Why does bootstrapping the residuals from a mixed effects model yield anti-conservative confidence intervals?

I typically deal with data where multiple individuals are each measured multiple times in each of 2 or more conditions. I have recently been playing with mixed effects modelling to evaluate evidence for differences between conditions, modelling individual as a random effect. To visualize uncertainty regarding the predictions from such modelling, I have been using bootstrapping, where on each iteration of the bootstrap both individuals and observations-within-conditions-within-individuals are sampled with replacement and a new mixed effect model is computed from which predictions are obtained. This works fine for data that assumes gaussian error, but when the data are binomial, the bootstrapping can take a very long time because each iteration must compute a relatively compute-intensive binomial mixed effects model.

A thought I had was that I could possibly use the residuals from the original model then use these residuals instead of the raw data in the bootstrapping, which would permit me to compute a gaussian mixed effect model on each iteration of the bootstrap. Adding the original predictions from the binomial model of the raw data to the bootstrapped predictions from residuals yields a 95% CI for the original predictions.

However, I recently coded a simple evaluation of this approach, modelling no difference between two conditions and computing the proportion of times a 95% confidence interval failed to include zero, and I found that the above residuals-based bootstrapping procedure yields rather strongly anti-conservative intervals (they exclude zero more than 5% of the time). Furthermore, I then coded (same link as previous) a similar evaluation of this approach as applied to data that is originally gaussian, and it obtained similarly (though not as extreme) anti-conservative CIs. Any idea why this might be?

• hm, I just noticed that in the data generating code for both cases I didn't actually add any between-individual variability that one is typically interested in eliminating by modelling individuals as random effects. I'll see if adding this variability changes the outcome; back in a few hours... Commented Jul 4, 2011 at 3:25
• If I remember correctly, bootstrap makes the estimate closer to the true population estimate. It doesn't say anything about the confidence interval. (cf. Kesar Singh, On asymptotic accuracy of Efron's bootstrap. Ann. Statist., 1981, 9, 1187-1195) Commented Jul 4, 2011 at 8:54
• @me: I can confirm that the addition of between-individuals variability in the data generation function does not improve performance of the bootstrap. I've uploaded the code I used to confirm this to the gist linked in the original post. Commented Jul 4, 2011 at 10:57
• @suncoolsu: I'm pretty sure that bootstrapped confidence intervals have been standard for quite some time. Efron mentions them in his 1978 paper describing the bootstrapping procedure generally, then he had a bunch of papers in the 80s and 90s on tweaks of the bootstrap procedure for more accurate confidence intervals (bias correction, accelleration, studentizing, etc). Commented Jul 4, 2011 at 11:27
• Have you read the following paper by Morris: "The BLUPs are not the best when it comes to bootstrapping". It may relate to your work. link Commented Aug 14, 2012 at 20:39