I am new to the permutation test and has found some online resources to learn how to conduct the permutation test. But I have yet to find a good explanation of how to find the power of a permutation test. Can someone explain this to me step-by-step or direct me to a good resource?

For example, suppose I want to apply the permutation test to the following hypothesis that can be checked using a normal distribution. Suppose there are two populations, from each population j=1,2 the following elements are drawn

    (Xj1, Xj2, ..., XjN)

I want to test the Null hypothesis

    mean(X1i) = mean(X2i)

against the alternative hypothesis that the two means are not equal.

For the power analysis, I want to find the power of the test if the true difference is K.

How would I actually do the power analysis in this case?

  • $\begingroup$ There does not appear to be any special feature or difficulty in computing the power of a permutation test: it works the same as computing the power of any other test. Could you articulate some aspect of this situation that you think merits a special discussion? $\endgroup$ – whuber Apr 12 '18 at 13:54
  • $\begingroup$ @whuber the way that I compute the power of a z-test or a t-test relies on the functional form of normal distribution. Since the permutation test is non-parametric test there doesn't seem to be an obvious equivalence. So I guess my question is just how to compute the power of a permutation test? $\endgroup$ – Amazonian Apr 13 '18 at 2:39
  • 1
    $\begingroup$ You have to begin by formulating a definite alternative hypothesis. After all, that's how power is defined: for each state of nature in the alternative hypothesis, it gives the expected true negative rate of the test. $\endgroup$ – whuber Apr 13 '18 at 14:36
  • $\begingroup$ @whuber I added an example with a definite alternative hypothesis. Can you please explain how to find the power of a permutation test using this example? $\endgroup$ – Amazonian Apr 16 '18 at 8:32
  • $\begingroup$ For a power calculation to be possible, you must be able to compute the sampling distribution of the test statistic for all possibilities allowed by the alternative hypothesis. It therefore behooves you to be as restrictive as possible, consistent with the context and objectives of your statistical setting. Unfortunately your alternative is too broad to permit any meaningful computation of power: because it doesn't specify anything about the distributions besides their means, the difference in means could be arbitrarily large and yet be undetectable with a permutation test. $\endgroup$ – whuber Apr 16 '18 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.