1
$\begingroup$

What do you think about using a permutation test each time the sample size is too small to meet the the assumptions of normality and homogeneity of variance etc ? Why do we use a non parametric wilcoxon test most of the time?

If there is any, could you please give me the pros of using a permutation test instead of a wilcoxon test ? can we say, since there is many repetitions and since an empirical distribution is generated with the permutation test, that the results are more "trustworthy" in presence of a small size (n=14) ?..

Context: I want to know if there is a significant different between two groups (placebo/treated group) but the distributions don't meet the normality assumption so i have to use a nonparametric test. I heard that permutation test is better than wilcoxon test why n is small but I don't get why? Is it because there is many repetition (permutations of the samples) in the permutation test and in the wilcoxon test there is not, so it give more "reality" to the result? or its just maybe because a permutation test is more powerful since the statistic usually used i just like the t-test ?...

If you have some sources, papers etc feel free to share !

Improve this question
$\endgroup$
9
  • $\begingroup$ Why not t-test? stats.stackexchange.com/q/121852/35989 $\endgroup$
    – Tim
    Commented Nov 3, 2022 at 16:47
  • 1
    $\begingroup$ Presumably because the OP seeks a guarantee on the type I error rate that is absent with a t-test in small samples. $\endgroup$
    – Glen_b
    Commented Nov 4, 2022 at 0:23
  • 1
    $\begingroup$ @learners A few comments on your post that could lead to clarifications (or relate to a potentially mistaken premise). 1. a permutation test is nonparametric. 2. a Wilcoxon test (by which I presume you mean a WIlcoxon signed rank test here) is a permutation test, performed on the signed ranks. 3. As for why you might prefer a signed rank test or a permutation test using some other statistic, this in part depends on what specific hypothesis you seek to test, (which you make no mention of), and potentially on what alternatives you might particularly look for power against. $\endgroup$
    – Glen_b
    Commented Nov 4, 2022 at 0:28
  • 1
    $\begingroup$ @learner I see now you meant either WIlcoxon test (only mentioning one sample size in the Q. threw me off). It sounds like you've got a slight misapprehension about what a permutation test is. A permutation test does not require random sampling of the permutation distribution - with small samples you can determine the complete permutation distribution (the thing you'd be sampling from if you used random sampling to undertake the exchanges of quantity that don't change the distribution under H0). With many rank-based tests you can derive a recurrence relation (simple in the case of ... ctd $\endgroup$
    – Glen_b
    Commented Nov 4, 2022 at 21:22
  • 1
    $\begingroup$ ctd ... the two WIlcoxon tests) to obtain the exact null distribution at middling sample sizes, and you also know various moments (e.g. the variance) of the test statistic under H0 and can rely on an asymptotic approximation in large samples (normal in the case of Wilcoxon tests). $\endgroup$
    – Glen_b
    Commented Nov 4, 2022 at 21:23

1 Answer 1

Reset to default
4
$\begingroup$

I think the answer to the question as to which test to use comes down to, Which hypothesis do you want to test ?

The Wilcoxon-Mann-Whitney test essentially tests if the values in one group tend to be larger in than the values in the other group.

In contrast, a permutation test on the means tests if there is a difference in the means.

You could also use a permutation test on the medians, or any other statistic of interest.

Start by deciding what you want to know, and then find an approach that addresses this. Note that your description of your goal: "I want to know if there is a significant different between two groups." is vague. There are many ways the values for those two groups could be different. (Mean, median, 90th percentile, ranked values, overall distribution...)

As general advice, be cautious of claims you encounter that you might phrase as, "I heard that permutation test is better than wilcoxon test why n is small". What's the source? Is it reliable? Was there any support or explanation?

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ You could even do a permutation test on the difference in mean rank . . . $\endgroup$
    – Glen_b
    Commented Nov 7, 2022 at 5:40
  • 2
    $\begingroup$ As an historical point, the Wilcoxon test was much easier to compute than a permutation test, when the Wilcoxon test was developed. It was easy to rank the values and calculate the sum of the ranks and look in the table. Even for modest samples, computing a permutation test was complicated without a computer. There are arguments from nonparametrics about choosing the scores to use, e.g., ranks for the Wilcoxon. One choice of scores gives the permutation test, which is best under certain conditions, possibly most. Permutation tests, now easily computed, could be more widely used. $\endgroup$
    – David Smith
    Commented Nov 7, 2022 at 15:00
  • 1
    $\begingroup$ books.google.de/books?id=nvDqDwAAQBAJ&pg=PA132 this book says on page 132 that the wilcoxon rank sum test is giving the same results as a permutation test computed with the rank sum statistic $\endgroup$
    – Ggjj11
    Commented Feb 19 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.