Resampling small datasets - Issue of overcounting?

Hypothetical situation here, as techniques like bootstrapping often fail for very small datasets.

Taking bootstrapping as an example nonetheless. we can easily compute the number of possible bootstrap (re)samples.

A nice answer is given by @whuber here:

Amount of Possible Bootstrap Samples

Suppose for a moment that bootstrapping is perfectly valid for low sample sizes. Now, say we have $$n = 5$$. From the posted solution above, it is found that there are a total of 126 possible boostrap resamples that can be drawn.

In bootstrapping, we usually take a large number of replications (10000 for instance). Taking this many replicates for such a small dataset, like the one above, seems bizarre, since resamples will be counted multiple times.

Question: Does this really matter? If yes. what is the effect on inference?

When bootstrapping, we are assuming that the sample is representative of the population.

The whole purpose of bootstrapping is to estimate a sampling distribution and infer the likely standard errors and confidence intervals for the population as a whole.

However, the issue with small sample sizes is that bias is more likely to exist relative to large samples - and (contrary to popular belief) bootstrapping does not fix this bias or remedy the issue of small sample sizes.

For instance, suppose one were to roll a dice five times. The numbers 4, 5, 6, 6, 6 are obtained, for an average of 5.4.

If one were to roll a dice a hundred times, an average closer to 3.5 could be expected - which is the theoretical mean.

However, small samples have a higher chance of deviating significantly from the population mean and so bootstrap sampling will not remedy this by virtue of simply generating more observations.

• Your last remark is incorrect, if we understand "resamples" to mean bootstrap samples of the data. The situation is one of sampling with replacement from a finite population (namely, of all possible bootstrap samples). When that sampling is performed uniformly and independently, (1) resamples are expected to recur after enough sampling is performed and (2) that will definitely not cause bias.
– whuber
Nov 26 '21 at 21:09
• @whuber: Taking the dice example again, if the five samples of 4, 5, 5, 6, 6 are continuously resampled, then surely the obtained bootstrap samples would still be biased as they would not be representative of all possible values from 1-6 that could be obtained? I am happy to be corrected if my understanding is wrong. Nov 26 '21 at 21:43
• Based on the first parts of this answer, Michael, I doubt your understanding is incorrect. I think there might be some miscommunication, though. When you refer to "resamples," are you talking about resamples of the original population or bootstrap samples? If they are bootstrap samples, they are intended to represent the data and they do not give a biased picture of the data. If you mean the former, then it would help your readers to make that a little clearer.
– whuber
Nov 26 '21 at 22:44
• The latter. Indeed, bootstrap samples produced from a sample of data will not give a biased picture of the data itself. However, if the original data sample is not representative of the population in the first instance, then neither will the bootstrap samples. I have edited the last sentence of my answer for clarity. Nov 26 '21 at 22:57
• Right. I didn't get that message when reading your last paragraph, because in this context "resamples" usually refer to resamples from the data. (+1 now.)
– whuber
Nov 26 '21 at 22:59

The issue with small sample sizes is not that you will repeat bootstrap samples but that the original small sample might not be so representative of the population.

Let’s obtain a sample of coin flips from a fair coin, so the true population is $$Binom(1, 0.5)$$, and let’s use your small sample size of $$n=5$$. In R…

set.seed(314) # For pi
x <- rbinom(5, 1, 0.5)

I get four $$0$$s (heads) and one $$1$$ (tails), which means a $$20\%$$ chance of tails, rather than the correct $$50\%$$. When we go to bootstrap this sample, we are telling the bootstrap procedure to sample from a $$Binom(1, 0.2)$$ distribution, which is quite a bit different from the true $$Binom(1, 0.5)$$ population.

When the sample size is larger, we are less likely to have a sample that is so drastically different from the population.

• Although it's obvious to me where the 0.2 (= 1 success in 5 trials) statistic comes from, it may not be the case for newcomers to statistics. Suggest possibly including a small edit explaining this more clearly. Though it could just be a Friday moment. Nov 26 '21 at 21:40

Many bootstrap replications are drawn in order to approximate (in a standard situation) the distribution of i.i.d. samples from the empirical distribution. Now if you can get this distribution explicitly, which is possible with a small sample as you can list all possible samples and their probabilities, it isn't necessary to approximate it by a much larger number of random bootstrap samples. Instead you can just use the full distribution of bootstrap samples (obviously potential problems with a lack of representativity of your small sample as mentioned in other answers still exist, but I believe this was not the question).

Note by the way that i.i.d. sampling from the empirical distribution will produce uniform probabilities over ordered rather than distinct samples, meaning that if you want to emulate the true bootstrap distribution by a set of samples, you will need some repetition of most of the 126 distinct samples. This would be approximated by randomly taking a very large number of bootstrap samples, so multiple counting of samples is not the issue here, rather what happens is you're using more computing power than necessary to do something less precise than possible with less computing effort.

• I can't follow your second paragraph at all. The bootstrap distribution is, by definition, the uniform distribution on all resamples (with replacement) from the empirical data. In light of this it's unclear what distinctions you are trying to draw between "ordered" and "distinct" samples and between "true" and "emulated" distributions.
– whuber
Nov 26 '21 at 22:49
• According to Efron and following literature, the bootstrap distribution is defined by i.i.d. sampling from the empirical distribution, which I believe is different. Let's discuss this here: stats.stackexchange.com/questions/461967/… Nov 27 '21 at 10:48