Suppose we have a test statistic designed for testing a null about two groups of samples.

If we apply permutation test to the groups of samples and the test statistic, will that change the null hypothesis?

What I heard of before is that permutation test is to test if the two groups of samples are coming from the same distribution, where "same" can be understood in various ways. So does it mean that after applying the permutation test, the original null will be changed?

An example, consider the case when the test statistic is to test if the locations of two groups of samples are the same.

  • the t-test tests if the mean of two normally distributed groups of samples have the same mean. Will applying permutation change its null, for example, from equal mean to equal distribution?

  • the Wilcoxon rank-sum test tests if the mean ranks of the two groups are the same. Will applying permutation change its null, for example, from equal mean rank to equal distribution?

What does permutation test bring to a location test?


2 Answers 2


Did you read this article which I pointed you to a discussion about? It talks about using a permutation test comparing means when the variances differ (and other cases) and shows that if the means are equal (on the population level) but the variances differ, there are cases where the type I error will not be what it is claimed to be.

This is not that hard of a process to simulate either, you could learn a lot for yourself by simulating the cases you are interested in and exploring the behavior of the tests.

This of course begs the question of whether comparing the means is even an interesting or appropriate question if the variances differ strongly. If you needed to lower your cholesterol and your doctor told you that 2 medicines for lowering cholesterol had the same mean amount dropped after 2 months on the medication, would you be happy taking either one? Or would you want to know that while the subjects taking the 1st medicine lowered their scores between 15 and 35 points while the majority of those in the second group actually raised their cholesterol score, but a few in the study had their scores plummet to extremely low levels (from which they later died)?

Or when comparing the average family income of 2 countries is it meaningful to say they have the same average family income where one country has a fairly even distribution of wealth and the other is mostly poverty with a couple of millionaire warlords?

  • $\begingroup$ Thanks! I read the paper. Do I understand correctly that permutation test applied to a location test doesn't test the null of location well, but rather tests whether the distributions of the two groups are the same in the sense of their locations? $\endgroup$
    – Tim
    Commented May 27, 2013 at 20:46

Yes. The permutation test approximates the null sampling distribution on the test statistic by the permutation operation. This is not the null under consideration in those other examples.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.