9
$\begingroup$

Is the purpose of permutation test to test the null that several groups of samples come from the same distribution?

I found its steps are

The steps in a permutation-based computation of the significance level of a test statistic are as follows:

i) Choose a test statistic, eg. a t-score for a comparison of two groups,

ii) Compute the test statistic for the gene of interest,

iii) Permute the labels on samples at random, and re-compute the test statistic for the rearranged labels; repeat for a large number (perhaps 1,000) permutations, and finally,

iv) Compute the fraction of cases in which the test statistics from iii) exceed the real test statistic from ii).

What kinds of test statistic should one choose in the first step?

The example uses the t-score, which measures the difference between two groups. But it seems to me that any statistic will work, not necessarily measuring the difference between two groups. Is it correct?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
11
$\begingroup$

Often there are several statistics that will all result in the same p-value/result. For example in a 2 sample case the difference of the 2 means, the mean of group A, and the sum of the values in group A will all result in the same p-value (this is because given the data values and sample sizes you can calculate the 1st 2 given only the 3rd). I would expect the t statistic to be similar to any of the above, but may not be exactly the same (due to the dividing by standard deviation(s)). There are other statistics that could be very different in the results, possibly the difference of the 2 medians, or the ratio of the 2 variances. These other statistics will be affected differently by the permutation process.

Your choice should be based on a combination of what is most interesting based on the science and question being asked (sometimes medians might be of more interest, other times means would be) and what will give you power to detect a difference in reasonable/meaningful alternatives. You can test this later by simulating data from cases that you think likely or interesting and watching how the statistics perform.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks, Greg! If the test statistic is not measuring difference between two groups, then does it still work technically? $\endgroup$
    – Tim
    May 21, 2013 at 16:36
  • 6
    $\begingroup$ Yes, Tim: you can perform a permutation test on literally any statistic. Part of being a good statistician is knowing how to choose a statistic that actually measures the effect of interest. A harder part is knowing how to choose a statistic that will do a good job of identifying differences when they do exist and of failing to identify "significant" differences when no differences (of any meaningful size) are present. If you make up some arbitrary statistic you can always do a permutation test and get a p-value, but the meaningfulness of this procedure may be questionable. $\endgroup$
    – whuber ♦
    May 21, 2013 at 16:53
  • 2
    $\begingroup$ @Tim, it depends on the statistic (and what you mean by measuring the difference). The sum of values in group A is not directly measuring a difference, but works. The ratio of variances might or might not be considered a difference and would probably not tell you much if the 2 distributions are just shifted in their centers (but identical variance and shape) but could be very meaningful in other cases. The maximum value in group A would probably not tell you much, even the difference in the 2 maximums is probably not a good choice most of the time. $\endgroup$
    – Greg Snow
    May 21, 2013 at 16:53
  • $\begingroup$ @Greg: Thanks! Is the purpose of permutation test only to test the null that several groups of samples come from the same distribution? $\endgroup$
    – Tim
    May 21, 2013 at 16:58
  • 2
    $\begingroup$ @Tim, the null hypothesis for a permutation test is that the distributions are identical, but there are many ways in which they can be non-identical, some more interesting than others, and the choice of test statistic will have more or less power for different types of non-identical. This post (tolstoy.newcastle.edu.au/R/e4/help/08/04/8485.html) and the others in the thread and the paper referenced may be of interest. $\endgroup$
    – Greg Snow
    May 21, 2013 at 17:40
3
$\begingroup$

You choose a test statistic that measures what you're interested in/has the properties you need.

If you want to compare means, you base it on differences of means; if you want a robust comparison of location, you measure something else; if you want to compare standard deviations, you use a statistic that does that; if you want to compare the whole distribution you use a statistic that compares distributions (such as a k-sample version of the Kolmogorov-Smirnov test, for an example).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.