Permutation tests: criteria to choose a test statistic

Question

I regularly use permutation tests and love their simplicity. I've learned most from the book "Resampling methods" by Good, in which the author seems quite creative in his choice of test statistics throughout the examples. Also this post gives the impression that there is a large freedom to choose a test statistic.

I do wonder if there are theoretical requirements a test statistic should comply with. Or can we just use any statistic as long as it intuitively makes sense and has good Type I/II error rates?

For example, when a permutation test is used instead of the t-test because of non-normal populations, I've seen a number of times that the permutation test p-value is still obtained from t-statistics. Although not necessarily wrong, it seems like a weird choice given the origin of the Student t distribution.

Answer 1

The t-statistic makes a lot of sense as a test statistic; many people find it intuitive. If I quote a t-statistic of 0.5 or 5.5, it tells you something - how many standard errors apart the means are.

The difficulty - at least with moderate non-normality - is not so much with using the statistic as using the t-distribution for its distribution under the null. The statistic is quite sensible.

Of course, if you expect substantially heavier tails than the normal, a more robust statistic would do better, but the t-statistic is not highly sensitive to mild deviations from normality (for example it's less senstive than the variance-ratio statistic).

If you want to use just the numerator of the statistic, that's great, it makes perfect sense as a permutation statistic, if you're interested in a difference in means. If you're interested in a more general sense of location shift, it opens up a plethora of other possibilities.

You're right to think there's a lot of freedom to chose a statistic and to tailor it to the particular circumstances - what alternatives you want power against, or what possible problems you'd like to be robust to (contamination, for example, can impact power).

There are really almost no restrictions - you're free to choose almost anything, including useless test statistics. There are some considerations that you really should think about when choosing tests, of course, but you're free not to.

--

That said, there are some criteria that can be applied in various circumstances.

For example, if you're particularly interested in a specific kind of hypothesis, you can make use of a statistic that reflects it - for example, if you want to test a difference in population means, it often makes sense to make your test statistic related to a difference in sample means.

If you know something about the kind of distribution you might have - heavy tails, or skew, or notionally light tailed but with some degree of contamination, or bimodal, ... you can devise a test statistic that might do well in such circumstances, for example, choosing a statistic that should perform well in the anticipated situation but has some robustness to contamination.

--

Simulation is one way to investigate power under various situations.