I have a treatment (T) and control (C) sample.

In my T sample, I have 600 observations, and in my C sample, I have 250 observations. Because the number of observations might be important for the issue I'm addressing, I wanted to test different subsampling bootstraps (X? bootstraps) of my T sample (force N=250). I can run a Wilcoxon rank-sum test for each bootstrapped T sample versus my C sample, yielding X p-values.

Why I'm downsampling

My scenario is similar to the following toy example. Imagine I have chosen 5 random numbers between 1 and 1,000. These will be target numbers. I then choose two sets of N random numbers (treatment and control test numbers), based on an unknown distribution, and calculate the distance to the nearest target number for each test number. I think the sample size, N, might be important in this case; therefore, I want to downsample.


Two Qs:
1) Is there a minimal number of bootstraps I should test (what X should I use)?
2) How do I deal with multiple p-values? How do I determine whether the difference between samples is statistically significant or not?

Any advice is much appreciated!

  • 1
    $\begingroup$ This doesn't make sense. What "might be important" about the number of treatment observations here, & why would downsampling help? I see no reason to do this. $\endgroup$ Aug 18, 2017 at 15:02
  • $\begingroup$ @gung I've edited my post. $\endgroup$
    – CPak
    Aug 18, 2017 at 15:41
  • $\begingroup$ This still doesn't make sense. Are you wondering which group will have the smallest minimum distance, or which will have the smallest average distance? $\endgroup$ Aug 18, 2017 at 16:05
  • $\begingroup$ @gung OK. I'm wondering if the median distance-to-nearest-target is meaningfully different between groups taking into account sample size. $\endgroup$
    – CPak
    Aug 18, 2017 at 16:12
  • 3
    $\begingroup$ By subsampling with different sample sizes than the original, you are doing just the opposite of taking sample size into account--you are forcibly making the issue go away, and thereby are losing power. The solution is to apply an appropriate test to the data you have, rather than studying samples of your data that have fewer values than the data you have. Therefore, it would be more constructive to change your question so it better describes your data, your objectives, and the context, rather than focusing on a controversial (and likely inferior) proposal. $\endgroup$
    – whuber
    Aug 19, 2017 at 16:23

1 Answer 1


I believe that you are applying the concept of bootstrapping wrongly.

What you are trying to do seems to be sampling randomly a couple of times 'in order to be able to perform a particular test, which you believe requires equal sample sizes' and then combine the p-values of those individual results into a single value.

A) First of all, you might be able to perform the Mann-Whitney(-Wilcoxon) U test (at least this is the case for your simplified number distance example). This test does not require the same sample sizes like the Wilcoxon test, which is performed on pairs.

B) Bootstrapping is more typically used to compute a statistic (and not for combining multiple randomly generated p-values).

  • Say you use have a statistic $W$ from your 600 treatment and 250 control observations (this could be some form of Wilcoxon test, or it could be the mean distance of the sample) and wish to estimate the distribution of this statistic (for comparisons or p-values) without being able to make good assumptions of the underlying distribution of the sampled variable,
  • then you can compute this distribution by using your 250 control observations to generate numerous random treatment samples of size 600 (sampling with replacement allows you to use your control sample of size 250 to generate treatment samples of size 600) as if they came from the same distribution as the control distribution,
  • and for these re-sampled distributions you can compute your statistic $W$ and generate a distribution for $W$ given the hypothesis that the treatment sample and the control sample have the same distribution (there are some technical issues that randomly re-sampling from the control sample is not really exactly the same as taking random samples from the distribution from which the control was taken).

So to answer your questions:

(2) You don't deal with p-values. With bootstrapping you generate multiple random cases of the statistic, and not multiple examples of the p-values. (and them you use the distribution of these randomly generated cases of the statistic to say something about differences or p-values)

(I do appreciate your idea how you tried to solve your perceived problem with the necessity of equal sample sizes to perform a test and still desiring to somehow use the power of your larger sample by using sub-sampling multiple times. In such case you might have used Fisher's combined probability test, although your p-values are not independent which might be an issue)

(1) The number of required bootstraps depends on the desired precision in the distribution of the statistic. Say you wish to calculate the 1st percentile of the distribution $W$, or a 98% confidence interval, then you need to repeat the bootstrapping (at least) 100 times and the higher the number of bootstraps the better your precision.

So if you do want to use bootstrapping then you use bootstrapping to find out the distribution of some statistic in your control sample (for instance a mean distance) and then compare this distribution with the value of the statistic of your treatment sample.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.