# Exhaustive Bootstrap Results

bootstrapping is the concept of taking many samples with replacement from the data to generate standard errors. Usually, we use random samples when the sample size is large as checking all possible samples would not be feasible. However, it can be done when the sample size is small. I tried it and got strange results. The exhaustive bootstrap produces results that are quite different from a regular bootstrap. As data I used this: x = [1,2,3,4].
Then I generated the standard error for the mean. The mean is 2.5. The exhaustive bootstrap looks like this:
[(1, 1, 1, 1), (1, 1, 1, 2), (1, 1, 1, 3), (1, 1, 1, 4), (1, 1, 2, 2), (1, 1, 2, 3), (1, 1, 2, 4), (1, 1, 3, 3), (1, 1, 3, 4), (1, 1, 4, 4), (1, 2, 2, 2), (1, 2, 2, 3), (1, 2, 2, 4), (1, 2, 3, 3), (1, 2, 3, 4), (1, 2, 4, 4), (1, 3, 3, 3), (1, 3, 3, 4), (1, 3, 4, 4), (1, 4, 4, 4), (2, 2, 2, 2), (2, 2, 2, 3), (2, 2, 2, 4), (2, 2, 3, 3), (2, 2, 3, 4), (2, 2, 4, 4), (2, 3, 3, 3), (2, 3, 3, 4), (2, 3, 4, 4), (2, 4, 4, 4), (3, 3, 3, 3), (3, 3, 3, 4), (3, 3, 4, 4), (3, 4, 4, 4), (4, 4, 4, 4)]
There are only 35 possible combinations. Now I take the mean of every resample, store it and then calculate the standard deviation of all results. The bootstrapped standard error is 0.7174300. However, the regular result is more like 0.5594 (for 5000 random resamples). Playing around shows me that the exhaustive bootstrap always overestimates the standard error. Why is this happening? My hunch is that the exhaustive bootstrap always includes some very rare combinations like (1,1,1,1), while this is very rare for a random sample. But according to the literature, the exhaustive bootstrap should work.

I think you want all permutations, not combinations. You have 1,1,1,2 but not 1,1,2,1 (or 1,2,1,1 or 2,1,1,1) and so that that sample mean (1.25) isn't represented four times as much as the sample mean of 1.0, which only shows up once for 1,1,1,1. If you use all 256 permutations you should see that the standard error is 0.56.

x_matrix <- matrix(data=NA, nrow=256, ncol=4)
x_matrix[, 1] <- rep(1:4, times = 1, length.out = NA, each = 64)
x_matrix[, 2] <- rep(1:4, times = 4, length.out = NA, each = 16)
x_matrix[, 3] <- rep(1:4, times = 16, length.out = NA, each = 4)
x_matrix[, 4] <- rep(1:4, times = 64, length.out = NA, each = 1)
sd(rowMeans(x_matrix))


 0.560112

• Thanks, very interesting! I wondered about that too, however, as for the mean, the ordering of the elements is without relevance, so I discarded that option. Maybe one could also solve this via weights as some solutions like (1,1,1,1) are unique, while others are equivalent, like (1,2,3,4) and (4,3,2,1). Thanks a lot. – unistata Apr 26 at 7:06