How do I perform a bootstrapped difference test when my bootstrapped sample sizes are smaller than the original sample sizes?

Typically I would do this by taking $p = \frac{\sum_{i=1}^{B} t_{i}^ {*} \geq t}{B}$, where

  • $t_{i}^ {*}=\frac{\bar{x^ {*} }-\bar{y}^ {*}}{\sqrt{\sigma^{*2}_x/m + \sigma^{*2}_y/m}}$ for a pair of bootstrapped samples $x^{*}$, $y^{*}$
  • $t=\frac{\bar{x}-\bar{y}}{\sqrt{\sigma^{2}_y/m + \sigma^{2}_x/m}}$

In this case, however, the obvious substitute for $t$ is

  • $t=\frac{\bar{x}-\bar{y}}{\sqrt{\sigma^{2}_x/n_1 + \sigma^{2}_y/n_2}}$, where $m \neq n_1 \neq n_2$

I am unsure if this is valid. Any help on this would be much appreciated!


I'm using bootstrapping to estimate if the respective population parameters for two populations $P_1$ and $P_2$ are significantly different.

For this purpose, I have samples from each population; namely, a sample $x_1$ of 900,000, and a sample $x_2$ of size 600,000. From these, I have done the following:

  • I've computed test statistics $\bar{x_1}$ and $\bar{x_2}$¹, as well as their difference
  • I've bootstrapped 2000 $x^∗_{1,i}$ 500,000-item samples and 2000 $x^{∗}_{2,i}$ 500,000-item samples
  • I've computed 2000 test statistics $\overline{x^∗_{1,i}}$ and $\overline{x^∗_{2,i}}$, as well as the differences between each

¹These are not the means, but it is convenient to represent them this way. Anyone is free to change this if it is annoying.

  • 1
    $\begingroup$ Why are you only drawing 500K samples per group, rather than the true sample sizes of 900K and 600K? $\endgroup$
    – Eoin
    Sep 21, 2022 at 11:08
  • $\begingroup$ @Eoin Memory reasons. Computing the statistic for a given 500k-item sample runs a /lot/ faster on my machine. Beyond that risks going over 28GB of RAM. $\endgroup$ Sep 21, 2022 at 11:11
  • $\begingroup$ It's worth noting that it takes about a couple months' worth of computation time to acquire these as-is $\endgroup$ Sep 21, 2022 at 11:19
  • $\begingroup$ Wow, ok. Out of curiosity, can you say what the statistic is? I suspect the approach you've proposed overestimates the uncertainty involved, but how to deal with that is beyond me. $\endgroup$
    – Eoin
    Sep 21, 2022 at 11:37
  • $\begingroup$ It's the optimal value for one of the weights in a neural network when that network is acting on the sample in question. $\endgroup$ Sep 21, 2022 at 11:38

1 Answer 1


Boostrap is used for assessing the uncertainty of the statistic. We know that the uncertainty changes with increasing sample size (approximately by $\sqrt n$), so if you use bootstrap samples that are smaller than the size of the data, you are overestimating the uncertainty. By using a smaller bootstrap sample size you can be almost certain that the result would be off, just how much off would depend on that particular scenario.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.