Permutation tests (also called a randomization test, re-randomization test, or an exact test) are very useful and come in handy when the assumption of normal distribution required by for instance,
t-test is not met and when transformation of the values by ranking of the non-parametric test like
Mann-Whitney-U-test would lead to more information being lost. However, one and only one assumption should not be overlooked when using this kind of test is the assumption of exchangeability of the samples under the null hypothesis. It is also noteworthy that this kind of approach can also be applied when there are more than two samples like what implemented in
coin R package.
Can you please use some figurative language or conceptual intuition in plain English to illustrate this assumption? This would be very useful to clarify this overlooked issue among non-statisticians like me.
It would be very helpful to mention a case where applying a permutation test doesn't hold or invalid under the same assumption.
Supppose that I have 50 subjects collected from the local clinic in my district at random. They were randomly assigned to received drug or a placebo at 1:1 ratio. They were all measured for paramerter 1
Par1 at V1 (baseline), V2 (3 months later), and V3 (1 year later). All 50 subjects can be subgrouped into 2 groups based on feature A; A positive = 20 and A negative = 30. They can also be subgrouped into another 2 groups based on feature B; B positive = 15 and B negative = 35.
Now, I have values of
Par1 from all subjects at all visits. Under the assumption of exchangeability, can I do comparison between levels of
Par1 using permutation test if I would:
- Compare subjects with drug with those received placebo at V2?
- Compare subjects with feature A with those having feature B at V2?
- Compare subjects having feature A at V2 with those having feature A but at V3?
- By which situation this comparison would be invalid and would violate the assumption of exchangeability?