20
$\begingroup$

It's often stated that permutation tests have no assumptions, however this is certainly not true. For example if my samples are somehow correlated, I can imagine that permuting their labels would not be the correct thing to do. Only think I found about this problem is this sentence from wikipedia: "An important assumption behind a permutation test is that the observations are exchangeable under the null hypothesis." Which I don't understand.

What are the assumptions of permutation tests? And how are these assumptions connected to different possible permutation schemes?

$\endgroup$
  • 4
    $\begingroup$ (+1) The Wikipedia quotation, although correct, is kind of funny, because when you get through the (obscuring) technical jargon it comes down to saying you should permute exactly those observations that you assume you can permute. $\endgroup$ – whuber Dec 17 '14 at 16:55
  • 1
    $\begingroup$ Difficult to answer since there are so many different permutation tests. E.g. for a k-sample comparison, heteroscedasticity between groups would violate exchangeability assumption. $\endgroup$ – Michael M Dec 17 '14 at 17:05
  • 1
    $\begingroup$ (+1) Based on Rubin (2015) "Causal Inference for Statistics", when the label (or treatment) is independent of the potential outcomes, then you can use permutation test.The logic is for each subject, it has two potential outcomes under label A and B, they are fixed. The label assignment procedure is random and if it is independent of the potential outcome, then you can think of performing this label assignment procedure N times, N is the total number of permutations of the labels, then you can get a distribution of the statistic you care about. Then you check quantile of the observed quantity $\endgroup$ – KevinKim Jan 13 '17 at 17:02
17
$\begingroup$

The literature distinguishes between two types of permutations tests: (1) the randomization test is the permutation test where exchangeability is satisfied by random assignment of experimental units to conditions; (2) the permutation test is the exact same test but applied to a situation where other assumptions (i.e., other than random assignment) are needed to justify exchangeability.

Some references regarding the naming conventions (i.e., randomization vs permutation): Kempthorne & Doerfler, Biometrika, 1969; Edgington & Onghena, Randomization Tests, 4th Ed., 2007

For assumptions, the randomization test (i.e., Fisher's randomization test for experimental data) only requires what Donald Rubin refers to as the stable unit treatment value assumption (SUTVA). See Rubin's 1980 comment on Basu's paper in JASA. SUTVA is also one of the fundamental assumptions (along with strong ignorability) for causal inference under the Neyman-Rubin potential outcomes model (cf. Paul Holland's 1986 JASA paper). Essentially, SUTVA says that there is no interference between units and that the treatment conditions are the same for all recipients. More formally, SUTVA assumes independence between the potential outcomes and the assignment mechanism.

Consider the two-sample problem with participants randomly assigned to a control group or a treatment group. SUTVA would be violated if, for example, two study participants were acquainted and the assignment status of one of them exerted some influence on the outcome of the other. This is what is meant by no interference between units.

The above discussion applies to the randomization test wherein participants were randomly assigned to groups. In the context of a permutation test, SUTVA is also necessary, but it may not rest on the randomization because there was none.

In the absence of random assignment, the validity of permutation tests may rely on distributional assumptions like identical shape of distribution or symmetric distributions (depending on the test) to satisfy exchangeability (see Box and Anderson, JRSSB, 1955).

In an interesting paper, Hayes, Psych Methods, 1996, shows through simulation how Type I error rates may become inflated if permutation tests are used with non-randomized data.

$\endgroup$
-5
$\begingroup$

See "A Primer on Quantitized Data Analysis and Permutation Testing" (page 88).

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • 1
    $\begingroup$ (-) Giving a citation is not an answer. First of all, someone could not have the book you quote. Second, you do not even provide a valid reference so we can only guess what do you quote! $\endgroup$ – Tim Dec 17 '14 at 17:05
  • 1
    $\begingroup$ I looked up that paper (it is freely accessible, btw: thank you!). However, it does not answer the question. All it states of relevance on p. 88 is "Permutation testing does not rely on traditional distribution assumptions; rather, it relies on R. A. Fisher’s (1935/1990) randomization model." We would appreciate it if you felt inclined to elaborate on this in an expanded answer. $\endgroup$ – whuber Dec 17 '14 at 17:28
  • $\begingroup$ The paper gives references where you can go to find permutation testing assumptions. Sorry but I don't have time to look those up for the person who asked the question. $\endgroup$ – Davester Dec 18 '14 at 17:31
  • $\begingroup$ Then you didn't actually answer the question. $\endgroup$ – neuronet Apr 14 '17 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.