3
$\begingroup$

Suppose we have a test statistic designed for testing a null about two groups of samples.

If we apply permutation test to the groups of samples and the test statistic, will that change the null hypothesis?

What I heard of before is that permutation test is to test if the two groups of samples are coming from the same distribution, "same" maybe in different sense. So does it mean that after applying the permutation test, the original null will be changed?

An example, consider the case when the test statistic is to test if the locations of two groups of samples are the same.

  • the t test statistic tests if the mean of two normally distributed groups of samples have the same mean. Will applying permutation change its null, for example, from equal mean to equal distribution?

  • the Wilcoxon rank-sum test test if two groups of samples have the mean ranks of the two groups are the same. Will applying permutation change its null, for example, from equal mean rank to equal distribution?

What does permutation test bring to a location test?

Thanks and regards!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Did you read this article which I pointed you to a discussion about? It talks about using a permutation test comparing means when the variances differ (and other cases) and shows that if the means are equal (on the population level) but the variances differ, there are cases where the type I error will not be what it is claimed to be.

This is not that hard of a process to simulate either, you could learn a lot for yourself by simulating the cases you are interested in and exploring the behavior of the tests.

This of course begs the question of whether comparing the means is even an interesting or appropriate question if the variances differ strongly. If you needed to lower your cholesterol and your doctor told you that 2 medicines for lowering cholesterol had the same mean amount dropped after 2 months on the medication, would you be happy taking either one? Or would you want to know that while the subjects taking the 1st medicine lowered their scores between 15 and 35 points while the majority of those in the second group actually raised their cholesterol score, but a few in the study had their scores plummet to extremely low levels (from which they later died)?

Or when comparing the average family income of 2 countries is it meaningful to say they have the same average family income where one country has a fairly even distribution of wealth and the other is mostly poverty with a couple of millionaire warlords?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I read the paper. Do I understand correctly that permutation test applied to a location test doesn't test the null of location well, but rather tests whether the distributions of the two groups are the same in the sense of their locations? $\endgroup$
    – Tim
    May 27, 2013 at 20:46
4
$\begingroup$

Yes. The permutation test approximates the null sampling distribution on the test statistic by the permutation operation. This is not the null under consideration in those other examples.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.