6
$\begingroup$

I want to find out if there is a calculation that can give me the minimum sample size to achieve a certain statistical power for a Permutation Test. I'm not assuming any parametric distribution for populations from where the samples come from.

I've been searching but haven't been able to find a satisfying answer. Statistical power seems like an overlooked aspect on most texts on permutation tests.

$\endgroup$
  • 1
    $\begingroup$ To compute a power you need an alternative specific enough to calculate a rejection rate. $\endgroup$ – Glen_b -Reinstate Monica Nov 25 '16 at 4:12
2
$\begingroup$

You partially answered your own question. Consider the reason you're often performing a permutation test. It's usually in circumstances where you have little faith in any particular parametric distribution or for some other reason want a non-parametric solution. In that case, how does one estimate power? You could be doing the permutation test in situations where the populations have multimodal distributions, uniform distributions, any combination for your conditions that don't match, etc. All of those might have different power functions.

Keep in mind that the way power is estimated requires an assumption of some world where you know effect size and distribution of the effect. Since you have no notion of the latter you can't estimate power a priori. The best you could do is estimate power for some particular distribution you might think is close, subset of likely distributions, or taking your data as the population. But the latter leaves you know way to get to power other than post hoc.

$\endgroup$
  • $\begingroup$ Thanks for your answer, it makes perfect sense. In my particular application, I use the permutation test to compare the difference in means from two populations with non-parametric distributions. I'm wondering if by knowing this, and by defining a priori the desired effect size to observe, I'll be able to calculate the necessary sample size. $\endgroup$ – intrepidSpaceman Nov 25 '16 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.