I want to find out if there is a calculation that can give me the minimum sample size to achieve a certain statistical power for a Permutation Test. I'm not assuming any parametric distribution for populations from where the samples come from.

I've been searching but haven't been able to find a satisfying answer. Statistical power seems like an overlooked aspect on most texts on permutation tests.

  • 1
    $\begingroup$ To compute a power you need an alternative specific enough to calculate a rejection rate. $\endgroup$
    – Glen_b
    Nov 25, 2016 at 4:12

2 Answers 2


You partially answered your own question. Consider the reason you're often performing a permutation test. It's usually in circumstances where you have little faith in any particular parametric distribution or for some other reason want a non-parametric solution. In that case, how does one estimate power? You could be doing the permutation test in situations where the populations have multimodal distributions, uniform distributions, any combination for your conditions that don't match, etc. All of those might have different power functions.

Keep in mind that the way power is estimated requires an assumption of some world where you know effect size and distribution of the effect. Since you have no notion of the latter you can't estimate power a priori. The best you could do is estimate power for some particular distribution you might think is close, subset of likely distributions, or taking your data as the population. But the latter leaves you know way to get to power other than post hoc.

  • $\begingroup$ Thanks for your answer, it makes perfect sense. In my particular application, I use the permutation test to compare the difference in means from two populations with non-parametric distributions. I'm wondering if by knowing this, and by defining a priori the desired effect size to observe, I'll be able to calculate the necessary sample size. $\endgroup$ Nov 25, 2016 at 15:47

I would wager permutation tests inflate the false negative rate in a small sample. Here's an extreme example to illustrate, but this applies with less extreme examples too:

  • r=1 with sample size 4
  • there are 4! = 24 permutations

Therefore: at least 1/24 ( = .042) permutations will have r=1 so p(r=1) >= 0.42. This is far greater than the real p-value of r=1, which is practically 0. The worst part is, with a p-threshold of .01, this result is not significant. Which is crazy; r=1 is the most significant possible.

This is because when you shuffle among existing values, you are not getting a true null distribution. The existing values are shuffled in order but not value. The values themselves are quantized. The values contributed to significance. This can, and will, bias the "null" shuffled distribution to something more significant. This holds for r-values less extreme than 1, and should be true for models other than correlation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.