# Small sample size: Permutation test or Wilcoxon test?

What do you think about using a permutation test each time the sample size is too small to meet the the assumptions of normality and homogeneity of variance etc ? Why do we use a non parametric wilcoxon test most of the time?

If there is any, could you please give me the pros of using a permutation test instead of a wilcoxon test ? can we say, since there is many repetitions and since an empirical distribution is generated with the permutation test, that the results are more "trustworthy" in presence of a small size (n=14) ?..

Context: I want to know if there is a significant different between two groups (placebo/treated group) but the distributions don't meet the normality assumption so i have to use a nonparametric test. I heard that permutation test is better than wilcoxon test why n is small but I don't get why? Is it because there is many repetition (permutations of the samples) in the permutation test and in the wilcoxon test there is not, so it give more "reality" to the result? or its just maybe because a permutation test is more powerful since the statistic usually used i just like the t-test ?...

If you have some sources, papers etc feel free to share !

• Why not t-test? stats.stackexchange.com/q/121852/35989
– Tim
Nov 3, 2022 at 16:47
• Presumably because the OP seeks a guarantee on the type I error rate that is absent with a t-test in small samples. Nov 4, 2022 at 0:23
• @learners A few comments on your post that could lead to clarifications (or relate to a potentially mistaken premise). 1. a permutation test is nonparametric. 2. a Wilcoxon test (by which I presume you mean a WIlcoxon signed rank test here) is a permutation test, performed on the signed ranks. 3. As for why you might prefer a signed rank test or a permutation test using some other statistic, this in part depends on what specific hypothesis you seek to test, (which you make no mention of), and potentially on what alternatives you might particularly look for power against. Nov 4, 2022 at 0:28
• @learner I see now you meant either WIlcoxon test (only mentioning one sample size in the Q. threw me off). It sounds like you've got a slight misapprehension about what a permutation test is. A permutation test does not require random sampling of the permutation distribution - with small samples you can determine the complete permutation distribution (the thing you'd be sampling from if you used random sampling to undertake the exchanges of quantity that don't change the distribution under H0). With many rank-based tests you can derive a recurrence relation (simple in the case of ... ctd Nov 4, 2022 at 21:22
• ctd ... the two WIlcoxon tests) to obtain the exact null distribution at middling sample sizes, and you also know various moments (e.g. the variance) of the test statistic under H0 and can rely on an asymptotic approximation in large samples (normal in the case of Wilcoxon tests). Nov 4, 2022 at 21:23

I think the answer to the question as to which test to use comes down to, Which hypothesis do you want to test ?

The Wilcoxon-Mann-Whitney test essentially tests if the values in one group tend to be larger in than the values in the other group.

In contrast, a permutation test on the means tests if there is a difference in the means.

You could also use a permutation test on the medians, or any other statistic of interest.

Start by deciding what you want to know, and then find an approach that addresses this. Note that your description of your goal: "I want to know if there is a significant different between two groups." is vague. There are many ways the values for those two groups could be different. (Mean, median, 90th percentile, ranked values, overall distribution...)

As general advice, be cautious of claims you encounter that you might phrase as, "I heard that permutation test is better than wilcoxon test why n is small". What's the source? Is it reliable? Was there any support or explanation?

• You could even do a permutation test on the difference in mean rank . . . Nov 7, 2022 at 5:40
• As an historical point, the Wilcoxon test was much easier to compute than a permutation test, when the Wilcoxon test was developed. It was easy to rank the values and calculate the sum of the ranks and look in the table. Even for modest samples, computing a permutation test was complicated without a computer. There are arguments from nonparametrics about choosing the scores to use, e.g., ranks for the Wilcoxon. One choice of scores gives the permutation test, which is best under certain conditions, possibly most. Permutation tests, now easily computed, could be more widely used. Nov 7, 2022 at 15:00