I want to implement basic randomization tests as described in Ernst, 2004 Permutation methods for exact inference. I have observations of a single continuous variable for n individuals falling into k groups. I want to know if differences in mean score per group are greater than would be expected to occur by chance.

Consider the following data (n=106 observations, k=8 groups):


Following Ernst, 2004: 5.1 a test for differences between means is:

T_stat<-sum((obs_means$values^2)*group_size) #the test statistic
#shuffle the data and recalculate
for (i in 1:length(results)){
  draw<-sample(values,length(values),replace=FALSE) #no replacement!
T_stat>crit_val #are there significant differences between means?

The test statistic here ("T_stat") is apparently equivalent to the F-statistic in a one-way ANOVA, so these results indicate that there is a significant difference between means at p=0.05.

Now, we want to know where that difference lies. Following Ernst, 2004: 5.2 the (approximate!) test is:

#mean scores per group
#get pairwise differences
names(diffs)<-paste(pairnames[1,],"vs",pairnames[2,]) # add names

for (i in 1:length(results)){
#print pairs with differences greater than the empirical critical value (there aren't any)

Assuming I've done that right, we find no differences between means when making pairwise comparisons. Since the test statistic we calculated in the first part for the overall difference between means was only slightly above the 'critical value' (the 95th percentile of the empirical distribution of test statistics that we simulated) I would chalk this up to this not being an exact test and conclude there are no differences between groups.

My questions are (i) is this computation and interpretation correct? (ii) is there an existing function in R that does this, perhaps better? For example, I would use the XNomial package if these data were categorical.

  • 1
    $\begingroup$ The standard package in R for permutation tests is called coin. $\endgroup$
    – Michael M
    Jun 25, 2020 at 20:29

2 Answers 2


After reading the section on anova, my guess is that the T statistic is a quick approximation for the F statistic. It does not account for explained variance so I guess if the groups have different spreads, then it might be problematic.

In your example it works ok. You can see how it compares with a real F statistic further into my answer.

My suggestion here is that you can compute a lot of these using built-in R functions, for example we can calculate the Tstat like this:

> T_stat
[1] 358402

obs_T = sum(tapply(values,group,function(i)length(i)*mean(i)^2))
[1] 358402

We can wrap this up and calculate all at one go:

Perm_Tstat = replicate(1000,
group = factor(group)
Perm_Fstat = replicate(1000, anova(lm(values ~ sample(group)))[1,4])
obs_F = anova(lm(values ~ group))[1,4]

hist(Perm_Tstat,br=50,main="Perm Tstat");abline(v=obs_T,col="blue")
hist(Perm_Fstat,br=50,main="Perm Fstat");abline(v=obs_F,col="blue")

enter image description here

For the second part, according to the paper:

We determine Cα by calculating the largest difference in treatment means for each randomization and finding the 1 − α quantile of this distribution.

So the statistic is the biggest absolute difference between two pairs, effectively maximum - minimum:

Perm_diff <- replicate(1000,diff(range(tapply(values,sample(group),mean))))



And I get the same result no difference between means. This method does not take into account the variance. If you look at the data:

boxplot(values ~ group)

enter image description here

The spread in groups 4,6,7,8 are much smaller and these are the ones giving you the significant difference. We can try a pairwise.t.test:


    Pairwise comparisons using t tests with pooled SD 

data:  values and group 

  2     3     4     5     6     7    
3 1.000 -     -     -     -     -    
4 1.000 1.000 -     -     -     -    
5 1.000 1.000 1.000 -     -     -    
6 1.000 1.000 0.025 0.017 -     -    
7 1.000 1.000 1.000 1.000 1.000 -    
8 1.000 1.000 1.000 1.000 0.860 1.000

P value adjustment method: holm 

You can see there is a difference and this agrees with the anova. In conclusion I think you need to think more about whether the difference method is suitable for you are doing, and whether there's a better way to test your hypothesis

  • $\begingroup$ Thanks @StupidWolf. That's cleaner script! I've looked into the coin package which does basically this (the difference-in-means test) in a more general framework (and I think with some intelligent corrections). I'll post that approach ASAP. I appreciate your answer but I don't think I can "accept" it since it is more of a comment / elucidation than a clear response. Perhaps reformulate it ? $\endgroup$
    – antifrax
    Jul 23, 2020 at 1:08
  • $\begingroup$ Hi @antifrax, there was more than one question and I tried to answer one of them. You can post your answer, most likely I don't have the time to reformulate my answer. From your comment, i gather there's nothing fundamentally wrong. $\endgroup$
    – StupidWolf
    Jul 23, 2020 at 8:13
  • 1
    $\begingroup$ The relationship between the F statistic and Ernst's "T-statistic" is confusing (he calls them 'equivalent'). This may be because Ernst's statistic follows an F-distribution (as per your histograms above!), but it's not a ratio of variances. $\endgroup$
    – antifrax
    Oct 9, 2020 at 12:40

re (ii): the coin package offers oneway_test as general test of the equality of distributions between groups. In this case running


gives the Fisher-Pitman permutation test based on 10k iterations. But as alluded to by StupidWolf, there's a problem here given that the goal is to test for differences between means. The Fisher-Pitman test compares both the mean and the variance simultaneously; if either is different the test rejects the null of equal distributions, and you don't know which one it is. In these data, a significant result could just be picking up on unequal variances. See doi: 10.2466/pr0.94.1.189-194 for more.

However, we can get a test of different central tendencies using 'median_test' from the same package (which uses the same underlying function). This gives the Brown-Mood Median test, which I haven't found much documentation on. But it does test differences in medians only, so this seems like the right test to use from coin. To determine post-hoc differences, we can use the pairwiseMedianTest function from the rcompanion package. See here for an example by the package creator, and don't forget the correction for multiple testing.

Pleasingly, you get the same result (overall significant differences, located by post-hoc testing between one group and two others) with both the Fisher-Pitman and the Brown-Mood. This suggests that Ernst's 'T-statistic', if not incorrect, is at least lower power.

To work out why this might be so we need to define the relationship between the F test, Ernst's 'T statistic', and the permutation tests available in coin. If anyone feels like posting that, we're listening!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.