# Are observations independent in bootstrapped resamples?

The bootstrap is often used for nonparametric inference. However, in some cases, it is useful to bootstrap and then conduct parametric tests within each resample (optionally, see References, but this is not required reading in any way for the question).

For example, you have a single continuous variable. You resample using a classical bootstrap, i.e., sampling with replacement. Then, within each resampled dataset, you conduct a hypothesis test that assumes independent observations, such as a standard $t$-test. Because there are repeated observations in the dataset, I would think that we have non-independent observations: the observations are "clustered" in the sense that all resampled observations mapping onto the same observation in the original dataset are 100% correlated.

More formally, say $Y$ is standard normal. Let $Y^{*}_i$ be a random variable representing the resampled $Y$ for the $i^{th}$ observation in a resampled dataset. Let's check if independence holds. As the number of resamples becomes large, we have (with some abuse of notation):

$$f_{Y^{*}_i}( y ) = N(0,1)$$

But, for all $i,j$ that map onto the same observation in the original dataset from which we resampled, the conditional distribution is degenerate because of the repeated observations and the continuous nature of $Y$:

$$f_{Y^{*}_i | Y^{*}_j}( y_i | y_j ) = 1\{y_i = y_j\} \ne f_{Y^{*}_i}( y_i )$$

So independence appears not to hold, and I would expect inference that assumes independence to be invalid (probably anticonservative).

However, simulations (with code below) indicate that in fact, in the exact situation described above, in fact we have exactly nominal Type I error.

So does independence hold in bootstrapped samples or not? If not, why doesn't nonindependence compromise inference as it usually does? Do I not even understand the definition of "independent observations"?

### Edit

I found this great discussion of the definition of independent observations that supports what I came up with intuitively. Hence, my question still stands.

### Simulation

library(doParallel)
# set the number of cores
registerDoParallel(cores=8)

sim.reps = 250
n=50
boot.reps = 500

res = as.data.frame( foreach( i = 1:sim.reps, .combine=rbind ) %dopar% {

# generate original data under H0 for a 1-sample t-test
y = rnorm(n)

pvals = c()
for (i in 1:boot.reps) {

# classical bootstrap
ids = sample(1:n, replace=TRUE)
y.star = y[ids]

# t-test
pvals[i] = t.test(y.star)$p.value } # should be 0.05 sum(pvals < 0.05) / length(pvals) } ) names(res) = "rej" res$rej = as.numeric(res$rej) # equals 0.05 mean(res$rej)


### References

Romano, J. P., & Wolf, M. (2007). Control of generalized error rates in multiple testing. The Annals of Statistics, 1378-1408.

• Whether or not they are independent depends on which random variables you are considering to be underlying your samples (since independence is a property of random variables). If it is random variables modelling the whole random experiment from the beginning, i.e. drawing the samples and then redrawing from them, then the $X_i^*$ are not independent. If it is only the redrawing from the bootstrap distribution then they are. – Winkelried Apr 8 '18 at 11:47

Think of it this way:

You have a population of individuals. You select an individual at random from the population, measure his weight and return him back to the population. You then select a second individual at random from the population, measure his weight and return him to the population. You continue this process until you end up with a set of 10 measured weights.

Two of the 10 sampled individuals give you the same weight of 70kg. Would you conclude that the two observations of 70kg are dependent just because they are the same? Not at all - each selection/draw from the population is independent of the others (i.e. the fact that you get a weight of 70kg in the first selection/draw, does not influence in any way the result you may get in other draws).

Don't confuse the value of the draw with the random mechanism used to guarantee that the draws are independent of each other. Two independent draws can produce the same value - that doesn't alter the fact that they are independent.

Edit:

In a repeated measure study, you can select a random sample of subjects and then measure each subject several times on some outcome variable (e.g., blood pressure). The values of the outcomes corresponding to different subjects will be independent of each other. However, within each subject, the values of the outcome variable will likely not be independent of each other. That is because all of these values would be affected by a shared set of factors - some observed, some unobserved - corresponding to that subject (e.g., age, sex).

• (+1) I was wondering about this as well. This part seems critical: "Don't confuse the value of the draw with the random mechanism used to guarantee that the draws are independent of each other." Could you please elaborate? How does this square with what I assumed to be the formal definition of independent observations above? – half-pass Apr 8 '18 at 11:08
• Is there a direct counterpart to that critical statement of yours in the usual sampling context, outside of bootstrapping? For example, in a case in which we have clustered data due to, e.g., repeated measures? I think that would help me understand. – half-pass Apr 8 '18 at 11:17
• Thanks. I did read the other Q&A, but this is partly why I'm still confused -- does what I wrote about about independence, using the same definition as in the linked Q&A, not hold? It seems to contradict what you're saying here about the sampling mechanism. – half-pass Apr 8 '18 at 19:11

Independence is a property of a collection of random variables defined on the same probability space. Whether or not your bootstrap samples are independent depends on which random variables you are considering to be underlying your samples.

Consider the random experiment:

Toss a coin twice, write down the outcomes, and sample twice with replacement from the outcomes.

With $\Omega_1:=\{0,1\}$ and $\Omega_2:=\{1,2\}$ you can model this random experiment with random variables on the space $$\Omega_3 = \Omega_1 \times \Omega_1 \times \Omega_2 \times \Omega_2.$$ With the random variables $$X_1:\Omega_3\to \mathbb{R},\ (a,b,c,d)\mapsto a,$$ $$X_2:\Omega_3\to \mathbb{R},\ (a,b,c,d)\mapsto b,$$ $$X_1^*:\Omega_3\to \mathbb{R},\ (a,b,c,d)\mapsto X_c(a,b,c,d),$$ $$X_2^*:\Omega_3\to \mathbb{R},\ (a,b,c,d)\mapsto X_d(a,b,c,d).$$ Here $X_1^*$ and $X_2^*$ are not independent.

However, if you consider the random experiment:

Given the outcomes of your coin toss, sample twice with replacement

and you model $X_1^*$ and $X_2^*$ to be the first and second sample, then they are independent.

• @half-pass The sample space of $X_1$ and $X_2$ is indeed only ${0,1}$. The tuples of the form $(a,b,c,d)$ are the elements of the probability space $\Omega_3$ on which the random variables are defined. Random variables are functions defined on a probability space and they must be defined on the same probability space if a relation between them, i.e. independence, etc. is to be considered. Usually in statistics one only specifies the random variables and its relations and implicitly assumes that there is an underlying probability space on which they are defined, but here I defined ... – Winkelried Apr 8 '18 at 21:11
• ... the functions explicitly to show that independence depends on how you define the underlying process. – Winkelried Apr 8 '18 at 21:12
• @half-pass Those tuples contain all the information on what happened during the random experiment. $a$ is the outcome of the first sample, $b$ the one of the second, if $c$ is $1$ the first resample is the first sample, it $c$ is $2$ the first resample is the second sample, analogously for $d$, as can be seen from the definitions of the functions. Say you have the elementary event $(0,1,2,1)$, then $X_1$ is $0$, $X_2$ is $1$, $X_1^*$ which is $X_c$ which is $X_2$ (because $c$ is $2$ in this case) is therefore $1$, analogously $X_2^*$ is $0$. – Winkelried Apr 8 '18 at 21:12
• @half-pass In case it is not clear $X_c$ is either $X_1$ or $X_2$ depending on whether $c$ is $1$ or $2$. Same for $X_d$. – Winkelried Apr 8 '18 at 21:20
• @half-pass Regarding whether or not the assumptions of your test are violated I would say no but I would have to think about it to give you a definitive answer. It depends on which random variables the test is supposed to test. I think it's the random variables that model only the resampling from the bootstrap distribution which therefore are independent. – Winkelried Apr 8 '18 at 21:34

I'm the OP. The answers from others were very good, and I have accepted one of them. I am also answering here to unite what I have learned.

Yes. In this context, the observations are independent. I believe the key issue is whether we are conditioning on cluster membership.

Above, I tried to check whether independence holds from the definition. But the following statement was wrong:

$$f_{Y^{*}_i | Y^{*}_j}( y_i | y_j ) = 1\{y_i = y_j\} \ne f_{Y^{*}_i}( y_i )$$

because it holds only for those $i,j$ for which we know that they come from the same observation in the original dataset. Thus, I had neglected to condition on a critical piece of information. Let $R_i$ be the index of the row in the original dataset from which we drew $Y^{*}_i$. Then the correct way to state the above is:

$$f_{Y^{*}_i | Y^{*}_j, R_i, R_j}( y_i | y_j, r_i=r_j ) = 1\{y_i = y_j\} \ne f_{Y^{*}_i}( y_i )$$

However, without conditioning on $R_i, R_j$, the observation that $Y_j^{*}=y_j$ carries no information about $Y_j^{*}$ since we don't know that they came from the same row in the original data. Thus:

$$f_{Y^{*}_i | Y^{*}_j}( y_i | y_j ) = f_{Y^{*}_i}( y_i )$$

so independence holds.

### A heuristic analog

Here is an analog to the more familiar notions of non-independence arising from the sampling mechanism. A classic way for non-independence to arise is through cluster-sampling, such as sampling schools and then sampling students within schools. This sampling scheme effectively "fixes" or conditions on cluster membership because we choose schools and then choose students within schools.

The critical analog to the bootstrapping independence is that if we instead sampled directly from the population of all students (rather than sampling schools), then the observations are independent whether we draw students from the same school or not. That's because we are no longer "conditioning" on cluster. Analogously, in the boostrapping world, the with-replacement sampling mechanism means we aren't conditioning on cluster; every observation has an equal chance of being sampled.

(I think the analog to the first example regarding cluster-sampling would be if we first chose a sample of observations in the original dataset, and THEN repeated each chosen row a number of times to create the resample. That, I believe, would lead to non-independent observations.)