Imagine that you conducted a study about children educational achievements. You took a random sample of schools from some area and from each school one class was included in the study. You conducted analysis and now want to use bootstrap to obtain confidence intervals for your estimates. How to do it?
First, notice that your data is hierarchical, it has several levels: schools, classes within schools, and students within classes. Since there is only one class per school, so the second level is nonexistent in your data. We can assume that there are some similarities within schools and differences between schools. If there are similarities within schools then if you sampled pupils at random, not taking into consideration their school membership you could possibly destroy the hierarchical structure of your data.
In general, there are several options:
- sample students with replacement,
- sample whole schools with replacement,
- first sample schools with replacement and then sample students (a) with replacement, or (b) without replacement.
It appears that the first approach is the worst one. Recall that bootstrap sampling should somehow imitate the sampling process in your study and you were sampling schools rather than individual students. Choosing between (2) and (3) is more complicated, but hopefully you can find research papers considering this topic (e.g. Rena et al. 2010, Field and Welsh, 2007). Generally options (2) or (3b) are preferable as it seems that including too much levels of sampling with replacement leads to biased results. You can find more information about this topic also in books by Efron and Tibshirani (1994) and Davison and Hinkley (1997). Notice that we have similar problem with bootstrapping time-series data and in this case we also rather sample whole blocks of series (e.g. whole season if we assume seasonality) rather than individual observations because otherwise the time structure would get destroyed. In practice there is no one-size-fits-all solution but with complicated data structures you should choose such bootstrap sampling scheme that best fits your data and your problem and if possible use a simulation study to compare different solutions.
Davison, A.C. and Hinkley, D.V. (1997). Bootstrap Methods and Their Application. Cambridge.
Efron, B. and Tibshirani, R.J. (1994). An Introduction to the Bootstrap. CRC Press.
Ren, S., Lai, H., Tong, W., Aminzadeh, M., Hou, X., & Lai, S. (2010). Nonparametric bootstrapping for hierarchical data. Journal of Applied Statistics, 37(9), 1487-1498.
Field, C. A., & Welsh, A. H. (2007). Bootstrapping clustered data. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(3), 369-390.