# Obtaining and interpreting bootstrapped confidence intervals from hierarchical data

I'm interested in obtaining a bootstrapped confidence interval on quantity X, when this quantity is measured 10 times in each of 10 individuals.

One approach is to obtain the mean per individual, then bootstrap the means (eg. resample the means with replacement).

Another approach is to do the following on each iteration of the bootstrapping procedure: within each individual, resample that individual's 10 observations with replacement, then compute a new mean for that individual, and finally compute a new group mean. In this approach, each individual observed in the original data set always contribute to the group mean on each iteration of the bootstrap procedure.

Finally, a third approach is to combine the above two approaches: resample individuals then resample within those individuals. This approach differs from the preceding approach in that it permits the same individual to contribute multiply to the group mean on each iteration, though because each contribution is generated via an independent resampling procedure, these contributions may be expected to vary slightly from eachother.

In practice, I find that these approaches yield different estimates for the confidence interval (ex. with one data set, I find that the third approach yields much larger confidence intervals than the first two approaches), so I'm curious what each might be interpreted to represent.

Your first approach is about a between S CI. If you wanted to measure within S then that's the wrong approach.

The second approach would generate a within S CI that would only apply to those 10 individuals.

The last approach is the correct one for the within S CI. Any increases in the CI are because your CI is more representative of a CI that could be applied to the population instead of those 10 S's.

According to Davison and Hinckley ("Bootstrap methods and their application", 1997, Section 3.8), the third algorithm is conservative. They advocate a fourth approach: simply resampling the subjects.

• Interesting, I'll have to look that reference up. Are you sure you mean "fourth" approach? The first approach I list seems to describe "simply resampling the subjects". Commented Aug 24, 2010 at 11:17
• Yes, it does, but it describes resampling the subject means. D&H advocate resampling the subjects and fitting the original model. Commented Sep 6, 2010 at 23:00
• You also might like to see the recently-published: Ren, Shiquan , Lai, Hong , Tong, Wenjing , Aminzadeh, Mostafa , Hou, Xuezhang and Lai, Shenghan(2010) 'Nonparametric bootstrapping for hierarchical data', Journal of Applied Statistics, 37: 9, 1487 — 1498 Commented Sep 7, 2010 at 23:28
• @Mike: resampling the whole clsuter is what survey statisticians do in their bootstraps. That's indeed a different procedure that would only be equivalent to your "first" approach if (i) you are only estimating the mean, and (ii) the data are unweighted and balanced. See also citeulike.org/user/ctacmo/article/1334050, citeulike.org/user/ctacmo/article/1475866, citeulike.org/user/ctacmo/article/582039. Commented Aug 13, 2011 at 15:10
• To update this post with more recent literature see "Saravanan, V., Berman, G. J., & Sober, S. J. (2020). Application of the hierarchical bootstrap to multi-level data in neuroscience. Neurons, behavior, data analysis and theory, 3(5)". They implement cluster bootstrapping in different examples and come to the conclusion that the procedure performs better and more sensitive than procedures that do not account for nested structures. Commented May 22, 2023 at 11:50