Papers About Permutation Version of Welch's t-test

Question

Permutation tests seem to provide a promising alternative for the unpaired t-test, requiring fewer assumptions. However, the core assumption of the permutation test, exchangeability, implies homogeneity of variances. So, as for the standard Student's t-test, when the variances and the group sizes are not equal, the TypeI error of the permutation test can be inflated. The following R script demonstrates this

library(perm)
typeIerrors <- 0
reps <- 1000
n1 <- 10
sd1 <- 5

n2 <- 30
sd2 <- 1

for (i in 1:reps){

  group1 <- rnorm(n=n1,sd=sd1)
  group2 <- rnorm(n=n2,sd=sd2)
  
  permRes <- permTS(group1,group2)$p.value
  if (permRes<.05){
    typeIerrors <- typeIerrors+1
  }
}
cat(sprintf('Type I error rate: %.2f\n',typeIerrors/reps))

Assuming homogeneity of variances seems to be problematic in many cases. The Wikipedia page for the Welch's test (the t-test version, which does not assume homogeneity of variances) notes on this topic: "It is not recommended to pre-test for equal variances and then choose between Student's t-test or Welch's t-test. Rather, Welch's t-test can be applied directly and without any substantial disadvantages to Student's t-test as noted above."

A simple approach to making the permutation test robust to variance heterogeneity might be to use the Welch test statistic as the test statistic. I did some research on this and only found one very short, seemingly unreviewed paper about the topic (http://vetdergikafkas.org/uploads/pdf/pdf_KVFD_779.pdf). Do you know other articles discussing the permutation version of the Welch's t-test? Optimally, discussing why or why not it is a good approach.

I found other, seemingly more complex approaches for the problem of making permutation tests robust towards unequal variances (for example, 1, and 2).

Answer 1

There is no need to refer to the Welch's test specifically. The permutation test is an alternative to the general class of two-sample $t$-tests.

In exact tests, like the T-test with equal variance assumption, one must analytically find the closed form of the sampling distribution of the test-statistic under the null hypothesis. For the equal variance T-test, it follows the T-distribution with pooled $n$ degrees of freedom. Welch found an approximate solution to the Fisher-Behren's problem using the Satterthwaite degrees of freedom, and that became the T-test which bears his name.

With the permutation testing, there is no need to make assumption about homoscedasticity or even normality (there are some rather general regularity conditions). You state the hypothesis to be tested:

$$\mathcal{H}_0 : \mu_1 = \mu_2 $$ $$\mathcal{H}_1 : \mu_1 \ne \mu_2 $$

Then randomly permute the group label a large number of times: this distribution is a numerical approximation of the test statistic under the null.

Answer 2

I finally found the paper myself: https://www.sciencedirect.com/science/article/pii/S0167715297000436

According to this paper, the permutation version of the Welch test is preferable over the normal Welch test in many situations.