Suppose I have two populations A and B , with sizes $n_1$ and $n_2$ respectively, where both $n_1$ and $n_2$ are large (say, above 500).

I want to test that the values $x_1, \dots, x_{n_1}$ of A have a mean (or median) that is different from $y_1, \dots, y_{n_1}$ of B.

I could do a classic non-parametric test (e.g. Mann Whitney U or Wilcoxon rank sum), but the power of classic non-parametric tests increase with sample size. Thus, we could "reject the null", even if the median (or mean) of A and B differ by a tiny amount, simply due to the large sample size.

Would a bootstrap method be more "appropriate" in this big data scenario?

  • 2
    $\begingroup$ You might do better to identify what you see (a priori) as an important difference and if you must test at all, maybe consider something like an equivalence test. $\endgroup$
    – Glen_b
    Apr 30, 2015 at 1:36

2 Answers 2


Judging from your attitude to a significant result, which you seem keen to avoid, are you sure you really want to be performing a hypothesis test at all?

"Significantly different" doesn't mean an important difference; it simply means your sample gives you evidence against the null hypothesis. Whenever you're running a hypothesis test, extra power is a feature, not a bug. The purpose of a test is to determine whether or not your precious sample (always to some extent limited and costly) gives you convincing evidence against the null, so deliberately using a lower powered test is making less effective use of the available information — it's much like throwing valuable data away, since your new test would now only have the same power as your previous test would have done on a smaller subset of your data. There are often good reasons to opt for a lower power test, particularly if we are concerned whether certain assumptions are met, but "fear of rejection" isn't one of them.

If there truly is no difference between the two populations your samples are drawn from, then there's no sense in which you'll somehow automatically reject the null hypothesis just because of the higher power of the test (to detect a difference which doesn't actually exist). If you were to set a 5% nominal significance level and there really is no difference, then your Type I error rate (assuming $H_0$ is true, the proportion of the time that you reject $H_0$) will be 5%. Or at least, very close to 5%, depending on the nature of your test, whether its assumptions are met exactly or approximately, and whether the test statistic is discrete or continuous.

Of course, at large sample sizes there's a strong argument for using a lower significance level like 0.1%, as since you've noted non-parametric tests on large sample sizes are going to give you good power anyway (if the data are actually normally distributed, then in large samples the tests you mention will have $\frac{3}{\pi} \approx 95.5\%$ of the efficiency of a t-test). When there is such a small probability of Type II error if your null hypothesis is incorrect by any appreciable margin, you might as well reduce your probability of Type I error too, at the expense of a small increase in the probability of Type II error. For more on balancing Type I and II error rates, and the interpretation of significant results in large samples, see this post.

But if you don't want to run a hypothesis test because you only care about "important" differences, and don't want to detect an "unimportant" difference, then searching for a lower powered test is not the solution. Instead you should consider stepping outside the hypothesis testing framework — wouldn't a suitable confidence interval for the difference, or some other effect size, be more interesting to you? Then you could judge for yourself what the evidence says about the size of the effect, rather than focus on the strength of the evidence for the effect's mere existence. In very large samples you can generally safely estimate the confidence interval for a mean difference without having to resort to a bootstrap.

  • 1
    $\begingroup$ Alternatively, you could also just alter your hypothesis to be more meaningful, i.e. $H_o: \mu_A - \mu_B \leq 1, H_a: \mu_A - \mu_b > 1$ if you consider a difference of 1 to be the minimal important difference, for example. $\endgroup$
    – Cliff AB
    Sep 18, 2015 at 5:19
  • 1
    $\begingroup$ @CliffAB raises an important point (sorry I didn't see this comment until now), it's worth thinking about performing an equivalence test if what you care about is whether there is an "important" difference. $\endgroup$
    – Silverfish
    Jun 25, 2020 at 20:18

Resampling method can also provide a assumption free distributions of the difference of the means, which in turn may give you some idea of the effect size of the difference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.