Terminology: The terms permutation and randomisation tests often seem to be used interchangably, but in some cases they have distinct meanings - with the meanings not always consistent. In this post by randomisation test I mean a test where we randomly permute the observations in our sample(s) to recompute some test statistic, with the ultimate goal of comparing our observed statistic with the distribution of recomputed statistics and obtaining a simulated p-value. It will not be possible to compute every permutation so in practice we randomly shuffle our data a 'large' number of times.

Summary of question(s): I would like to know

  • What the "null hypothesis" of a randomisation (permutation) test should be, and how the results of the test should be interpreted? In particular focussing on an example where the two populations have the same mean, but different variances. Can we use this test to test for specific things like a difference in means, or can we only test for whether the distributions are different?

  • How does the choice of null hypothesis and statistic affect exchangability, if at all?

  • I would appreciate a walkthrough of how a randomisation test could be applied to the example I give below.

Motivating Example: (See related:1 2) Two populations with the same mean but different variances. Group A, 30 people taken from $N(0,1)$. Group B 10 people from $N(0,4)$. We use a randomisation (permutation) test to obtain a simulated p-value for the difference in means. We observe that the simulated p-value is less than 0.05 about 33% of the time.

Python Code

import numpy as np
# Let's shuffle the labels of the groups and calculate the new difference in means.
# We will discuss later whether this is valid and how to interpret this
# We will count how many times we get a "p-value" less than 0.05
small_pval = 0
# We will simulate taking samples from our two populations many times, and running a randomisation test on them.
for i in range(N):
    # Take samples from our two distributions
    A = np.random.normal(loc=0,scale=1,size=30)
    B = np.random.normal(loc=0,scale=6,size=10)
    observed_diff = np.mean(A)-np.mean(B)
    # Combine into one list
    combined = np.concatenate((A,B),0) 
    differences_in_means = np.array([])
    for i in range(1000):
        diff = np.mean(combined[:20]) - np.mean(combined[20:])
        differences_in_means = np.append(differences_in_means, diff)
    # How often do we see a difference as extreme as the one we observed initially
    # Take absolute value for 2 tail but I think this is not very significant - :D
    simulated_pvalue =  np.mean(np.abs(differences_in_means) >= np.abs(observed_diff))
    if simulated_pvalue < 0.05:
print("We found simulated p-values less than 0.05 ", 100*small_pval/N, "% of the time")

Running the code above, we would find simulated p-values <0.05, about 33% of the time. Now, the statistic we have 'tested' is difference in means, however we know that both distributions have the same mean. The first thing I am trying to grasp, is what does the simulated p-value actually mean, in particular what is the associated null hypothesis for these p-values?

I could say something like: My null hypothesis is that the mean of population A = mean population B, I permute the labels many times, and find that the difference in means I have observed is very unusual (simulated p-value is smaller <0.05) and so I reject my null hypothesis and conclude (wrongly) that the means of the two populations are actually different. As the simulation above shows, if I repeated this, I would make this mistake about 33%, much more than the 0.05% we might expect.

I think that part of the problem here is exchangability and the set up of the "null hypothesis". I think that here due to the difference in std deviation, I cannot say that the labelling of group A and B is completely arbitrary and exchange them however I wish. However I think the null hypothesis above is not right for the randomisation test, but I have commonly seen people use the test this way - essentially as a direct replacement for the Student or Welch t-test. What should the null hypothesis be here to use a randomisation test? These quotes from Howell 3 seem relevant:

"Our null hypothesis has nothing to do with parameters, but is phrased rather vaguely, as, for example, the hypothesis that the treatment has no effect on the how participants perform. That is why I earlier put "null hypothesis" in quotation marks. This is an important distinction. The alternative hypothesis is simply that different treatments have an effect. But, note that we haven't specified whether the difference will reveal itself in terms of means, or variances, or some other statistic. We leave that up to the statistic we calculate in running the test. That might be phrased a bit more precisely by saying that, under the null hypothesis, the score that is associated with a participant is independent of the treatment that person received"

""I need to say something about exchangeability. It applies to the null hypothesis sampling distribution--in other words, data are exchangeable under the null. Phil Good, for example, is a big fan of this term. He would argue that if the scores in one group have a higher variance than the other, then the data are not exchangeable and the test is not valid. BUT, if the hypothesis being tested is that treatments have no effect on scores, then under the null hypothesis why would one set of scores have a higher variance other than by chance? The problem is that we have to select the statistic to test with care. We normally test means, or their equivalent, but we also need to consider variances, for example, because that is another way in which the treatment groups could differ. If we are focussing on means, then we have to assume exchangeability including variance. But we need to be specific. So much for that little hobby horse of mine."

Howell also shares the following extract from Edgington (1986).

"Just as the reference set (read as "sampling distribution" for now) of data permutations is independent of the test statistics, so is the null hypothesis. A difference between means may be used as a test statistic, but the null hypothesis does not refer to a difference between means. The null hypothesis, no matter what test statistic is used, is that there is no differential effect of the treatments for any of the subjects. ... Thus the alternative hypothesis is that the measurement of at least one subject would have been different under one of the other treatment conditions. Inferences about means must be based on nonstatistical considerations; the randomization test does not justify them." (p. 531)"

Based on this, I wonder, if instead my "null hypothesis" was that the two populations are from the same distribution and then I have a small simulated p-value I could conclude that because the difference in means we have observed is so unusual the distributions of the two populations must be different. This also seems strange to me, we have made the correct conclusion that the populations are from different distributions where the statistic we are using is difference in means, even though the means are actually the same. I note also if this was the null hypothesis, then in this case we have very low power here (33%), and wonder if that is precisely because the statistic we are using is the difference in means.

If the 'correct' null hypothesis for a (two sample) randomisation test is "The two populations are the same" and the alternative is then "the two populations are not the same" would this mean:

  1. I have more freedom with respect to exchangability, for example we now have exchangability in the example above.
  2. The power (and type 1 error rate) of the permutation test may be quite sensitive to whatever statistic I decide to use (say difference in means or Welch's t).
  3. If we find a significant result using some statistic, say difference in means, all we can conclude from this is the two populations are different, but we actually cannot conclude anything more (like the means are different).

In the post here 1 , an example very similar to the above is considered. It is mentioned that the type 1 error rate is inflated, and this can be remedied by using the Welch t statistic instead of the difference in means. The problem that is mentioned is lack of exchangability.

  1. Why does the Welch t-statistic have a lower type 1 error rate than just looking at the difference in means?
  2. If the null hypothesis is "The means of these two distributions are the same", then does using Welch t-statistic mean we have exchangability, if so why? The answer of BruceET here says

"A permutation test with the Welch t statistic as metric treats samples with unequal variances as exchangeable (even if data may not be normal)" 3) Am I right in thinking if we want to specifically conclude something about the means, then we need to set up the null hypothesis carefully and choose our statistic carefully (to ensure exchangability), but if we just want to conclude the samples are from different populations then we have more freedom?

A few references to help highlight my confusion

The answer to this similar question:

"Strictly speaking, the null hypothesis is that the distributions are the same, not just that they have the same means. (If they had same means but difference variances, the test would have the wrong Type I error rate.)"

whereas the answer to this question phrases the null hypothesis of the permutation test in terms of means.

In this copy of ThinkStats the permutation test is introduced, it seems to me that text is saying we can choose whatever statistic we are interested in, and see if there is a significant difference in it, and then make a conclusion about that statistic. However we have seen from the example above that is not right, we found that we had an 'extreme' difference in means even though the means of the two groups are the same.

"One way to model the null hypothesis is by permutation; that is, we can take values for first babies and others and shuffle them, treating the two groups as one big group..."

"Choosing the best test statistic depends on what question you are trying to address. For example, if the relevant question is whether pregnancy lengths are different for first babies, then it makes sense to test the absolute difference in means, as we did in the previous section. If we had some reason to think that first babies are likely to be late, then we would not take the absolute value of the difference; instead we would use this test statistic"

This question, which seems relevant but I do not think addresses my exact questions, it does not mention exchangability nor how the test should be “set up” with a null and alternate hypothesis.

This question is closest. The asker posts a comment on the top answer which is one of the things I am getting at, but their comment is unanswered.

Thanks! I read the paper. Do I understand correctly that permutation test applied to a location test doesn't test the null of location well, but rather tests whether the distributions of the two groups are the same in the sense of their locations?

I am a novice statistician. As you can no doubt tell, I am having trouble even articulating my questions. If I have misrepresented anything in any of the links or quotes provided that is not my intention, it is just due to my own confusion, I would gladly accept any edits/comments. I have also tried to link to other relevant questions, I do not think my question is a duplicate, if I have overlooked something relevant I would be glad to have it brought to my attention.

  • $\begingroup$ This may partially answer your question: academic.oup.com/bioinformatics/article/22/18/2244/317881 $\endgroup$
    – David B
    Apr 23 at 14:19
  • 2
    $\begingroup$ A correct permutation test in this situation would take a dataset of $X_1,\ldots, X_m$ drawn iid from a zero-mean distribution with unit variance and, independently, $Y_1,\ldots,Y_n$ drawn from a zero-mean distribution with (known) variance $\sigma^2,$ and create a permutation distribution by defining $Z_i=Y_i/\sigma,$ permuting the vector $(X_1,\ldots,X_m,Z_1,\ldots,Z_n),$ and multiplying the last $n$ values by $\sigma.$ The permuted dataset is $(X^\prime_1,\ldots,X^\prime_m,Y^\prime_1,\ldots,Y^\prime_n).$ $\endgroup$
    – whuber
    May 22 at 12:40

2 Answers 2


It makes sense to separate the discussion of the null hypothesis to test from the discussion of the test statistic chosen.

The null hypothesis is generally the probability model based on which the test level, i.e., the rejection probability of the null hypothesis, is computed.

The permutation test in this example will create a situation in which you have the same distribution in both samples. The permutation principle will simulate the correct rejection probability if both distributions are assumed the same. So this is the null hypothesis. This obviously implies (but is not equivalent with) equality of the means (assuming they exist). Note that there are other permutation tests where the null hypothesis is different (e.g., independence between two variables).

The test statistic will define how you measure deviations from the null hypothesis, so it will implicitly define the alternative against which you are testing. For example, if you choose the mean difference, you will run the test in such a way that it will be particularly sensitive (i.e., have large power) against situations in which the means are different. So, on one hand one could say "you can use any test statistic you want" (if it is only about simulating a valid p-value by the permutation procedure), but on the other hand, you may want to choose a test statistic that has good power for detecting the specific deviations from equality of distributions you care about in the application of interest.

So you are basically testing equality of distributions against differences in means with the mean differences statistic. If the truth is in neither of these classes (such as different variances but same mean), anything can happen. (This somehow explains why there is a temptation to say that even the null hypothesis here is "equal means", but it isn't, as for equal means if distributions are different, the permutation test cannot control the level properly.)


Nice discussion, and yes, the culprit is the difference in variance for the two populations; that is why the data are not exchangeable.

I have often abserved that permutation tests for difference in means react very sensitive to differences in variance if the test statistic used is just the difference in sample means, in particular, when sample sizes are different. Using the Welch t statistic makes the test much more robust against difference in population variance. The problem of testing difference in means when variances are also different, but we are not interested in them, goes under the name Behrens-Fisher problem.

Here is a paper by A. Janssen (1997) that shows that the Welch statistic asymptotically gives the correct test size for permutation test, even when variances are different: Janssen 1997

You can find more followup papers on the topic on the net.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.