Permutation tests seem to provide a promising alternative for the unpaired t-test, requiring fewer assumptions. However, the core assumption of the permutation test, exchangeability, implies homogeneity of variances. So, as for the standard Student's t-test, when the variances and the group sizes are not equal, the TypeI error of the permutation test can be inflated. The following R script demonstrates this

typeIerrors <- 0
reps <- 1000
n1 <- 10
sd1 <- 5

n2 <- 30
sd2 <- 1

for (i in 1:reps){

  group1 <- rnorm(n=n1,sd=sd1)
  group2 <- rnorm(n=n2,sd=sd2)

  permRes <- permTS(group1,group2)$p.value
  if (permRes<.05){
    typeIerrors <- typeIerrors+1
cat(sprintf('Type I error rate: %.2f\n',typeIerrors/reps))

Assuming homogeneity of variances seems to be problematic in many cases. The Wikipedia page for the Welch's test (the t-test version, which does not assume homogeneity of variances) notes on this topic: "It is not recommended to pre-test for equal variances and then choose between Student's t-test or Welch's t-test. Rather, Welch's t-test can be applied directly and without any substantial disadvantages to Student's t-test as noted above."

A simple approach to making the permutation test robust to variance heterogeneity might be to use the Welch test statistic as the test statistic. I did some research on this and only found one very short, seemingly unreviewed paper about the topic (http://vetdergikafkas.org/uploads/pdf/pdf_KVFD_779.pdf). Do you know other articles discussing the permutation version of the Welch's t-test? Optimally, discussing why or why not it is a good approach.

I found other, seemingly more complex approaches for the problem of making permutation tests robust towards unequal variances (for example, 1, and 2).


There is no need to refer to the Welch's test specifically. The permutation test is an alternative to the general class of two-sample $t$-tests.

In exact tests, like the T-test with equal variance assumption, one must analytically find the closed form of the sampling distribution of the test-statistic under the null hypothesis. For the equal variance T-test, it follows the T-distribution with pooled $n$ degrees of freedom. Welch found an approximate solution to the Fisher-Behren's problem using the Satterthwaite degrees of freedom, and that became the T-test which bears his name.

With the permutation testing, there is no need to make assumption about homoscedasticity or even normality (there are some rather general regularity conditions). You state the hypothesis to be tested:

$$\mathcal{H}_0 : \mu_1 = \mu_2 $$ $$\mathcal{H}_1 : \mu_1 \ne \mu_2 $$

Then randomly permute the group label a large number of times: this distribution is a numerical approximation of the test statistic under the null.

  • 2
    $\begingroup$ As far as I understand, the core statement of this answer: "The permutation test is an alternative to the general class of two-sample 𝑡-tests." is wrong but a widely spread belief. The crux is in the "rather general regularity conditions", which contain exchangeability, which implies homoscedasticity. This leads to inflated Type I errors, when using a standard permutation test to test for mean differences on unequal sample size, unequal variance data. See, for example, academic.oup.com/bioinformatics/article/22/18/2244/317881. $\endgroup$ Apr 29 '19 at 15:20
  • $\begingroup$ @JulianKarls at a glance, this paper discusses the issues of using the pseudolikelihood, e.g. ignoring dependence structures (and quite severely with $\rho=0.9$). I have some issues with the writing style. Worth passing the question to another SE post where you could supply some code or findings? $\endgroup$
    – AdamO
    Apr 29 '19 at 16:16
  • $\begingroup$ I added a very small simulation study that shows the alpha inflation. $\endgroup$ Apr 29 '19 at 17:30
  • $\begingroup$ @JulianKarls Thanks for sharing. Yes that would cause a problem. Why don't you calculate the Welch's t-test test statistic in the permuted datasets? Maybe that's what you're asking in the question. I've always done it that way but I suppose I hadn't thought why. It corrects the $\alpha$ level in your simulation. $\endgroup$
    – AdamO
    Apr 29 '19 at 20:51

I finally found the paper myself: https://www.sciencedirect.com/science/article/pii/S0167715297000436

According to this paper, the permutation version of the Welch test is preferable over the normal Welch test in many situations.

  • $\begingroup$ Could you add more details? I cannot access the paper. $\endgroup$
    – user271536
    Jan 6 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.