21
$\begingroup$

In the literature the terms Randomization and Permutation are used interchangeably. With many authors stating "Permutation (aka randomization) tests", or vice versa.

At best I believe the difference is subtle, and it lies in their assumptions about the data and potential conclusions which can be drawn. I just need to check whether my understanding is correct, or whether there is a deeper difference that I am missing.

Permutation tests assume that the data is sampled randomly from an underlying population distribution (the population model). This means that the conclusions drawn from the permutation test are generally applicable to other data from the population [3].

Randomization tests (randomization model) "permit us to drop the implausible assumption of typical psychological research—random sampling from a specified distribution" [2]. However that means that the conclusions drawn are only applicable to the samples used in the test [3].

Surely though, the difference is only in terms of the definition of population. If we define the population to be 'all patients with the ailment and are suitable for treatment' then the permutation test is valid for that population. But because we've restricted the population to those which are suitable for treatment, it is really a randomization test.

References:
[1] Philip Good, Permutation Tests: A practical guide to resampling methods for testing hypotheses.
[2] Eugene Edgington and Patric Onghena, Randomization tests.
[3] Michael Ernst, Permutation Methods: A basis for exact inference

Improve this question
$\endgroup$
1
  • $\begingroup$ Is it so that using normal-theory based methods will allow one to conclude beyond sample (to population) whereas using randomisation methods will result in our conclusions being applicable only to the sample? $\endgroup$
    – info_seeker
    Dec 6, 2018 at 14:04

1 Answer 1

Reset to default
13
$\begingroup$

There is quite a bit of overlap and the most common form of the permutation test is a form of a randomization test.

Some purists consider the true permutation test to be based on every possible permutation of the data. But in practice we sample from the set of all possible permutations and so that is a randomization test.

There are also bootstrap tests, if we don't find every possible bootstrap sample but rather sample from the possible set (what is usually done) then this is also a randomization test (but not a permutation test).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.