Papers About Permutation Version of Welch's t-test

Question

Permutation tests seem to provide a promising alternative for the unpaired t-test, requiring fewer assumptions. However, the core assumption of the permutation test, exchangeability, implies homogeneity of variances. So, as for the standard Student's t-test, when the variances and the group sizes are not equal, the TypeI error of the permutation test can be inflated. The following R script demonstrates this

library(perm)
typeIerrors <- 0
reps <- 1000
n1 <- 10
sd1 <- 5

n2 <- 30
sd2 <- 1

for (i in 1:reps){

  group1 <- rnorm(n=n1,sd=sd1)
  group2 <- rnorm(n=n2,sd=sd2)
  
  permRes <- permTS(group1,group2)$p.value
  if (permRes<.05){
    typeIerrors <- typeIerrors+1
  }
}
cat(sprintf('Type I error rate: %.2f\n',typeIerrors/reps))

Assuming homogeneity of variances seems to be problematic in many cases. The Wikipedia page for the Welch's test (the t-test version, which does not assume homogeneity of variances) notes on this topic: "It is not recommended to pre-test for equal variances and then choose between Student's t-test or Welch's t-test. Rather, Welch's t-test can be applied directly and without any substantial disadvantages to Student's t-test as noted above."

A simple approach to making the permutation test robust to variance heterogeneity might be to use the Welch test statistic as the test statistic. I did some research on this and only found one very short, seemingly unreviewed paper about the topic (http://vetdergikafkas.org/uploads/pdf/pdf_KVFD_779.pdf). Do you know other articles discussing the permutation version of the Welch's t-test? Optimally, discussing why or why not it is a good approach.

I found other, seemingly more complex approaches for the problem of making permutation tests robust towards unequal variances (for example, 1, and 2).

Answer 1

There is no need to refer to the Welch's test specifically. The permutation test is an alternative to the general class of two-sample $t$-tests.

In exact tests, like the T-test with equal variance assumption, one must analytically find the closed form of the sampling distribution of the test-statistic under the null hypothesis. For the equal variance T-test, it follows the T-distribution with pooled $n$ degrees of freedom. Welch found an approximate solution to the Fisher-Behren's problem using the Satterthwaite degrees of freedom, and that became the T-test which bears his name.

With the permutation testing, there is no need to make assumption about homoscedasticity or even normality (there are some rather general regularity conditions). You state the hypothesis to be tested:

$$\mathcal{H}_0 : \mu_1 = \mu_2 $$ $$\mathcal{H}_1 : \mu_1 \ne \mu_2 $$

Then randomly permute the group label a large number of times: this distribution is a numerical approximation of the test statistic under the null.

Answer 2

You may be also interested in these papers:

1. https://projecteuclid.org/journals/annals-of-statistics/volume-41/issue-2/Exact-and-asymptotically-robust-permutation-tests/10.1214/13-AOS1090.full (free access)

page 8/24 linking to

JANSSEN , A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the general- ized Behrens–Fisher problem. Statist. Probab. Lett. 36 9–21. MR1491070

https://www.sciencedirect.com/science/article/abs/pii/S0167715297000436 saying that:

It is shown that permutation tests based on studentized statistics are asymptotically exact of size α also under certain extended non-i.i.d. null hypotheses. To demonstrate the principle the results are applied to the generalized two-sample Behrens-Fisher problem for testing equality of the means under general non-parametric heterogeneous error distributions. Within this setting we propose a permutation version of the Welch test which is an extension of Pitman's two-sample permutation test. These results are special cases of a conditional central limit theorem for studentized permutation statistics which also applies to asymptotic power functions.

2. https://link.springer.com/article/10.3758/s13428-021-01595-5 (free access)

Noguchi, K., Konietschke, F., Marmolejo-Ramos, F. et al. Permutation tests are robust and powerful at 0.5% and 5% significance levels. Behav Res 53, 2712–2724 (2021). https://doi.org/10.3758/s13428-021-01595-5

Abstract:

Recent replication crisis has led to a number of ad hoc suggestions to decrease the chance of making false positive findings. Among them, Johnson (Proceedings of the National Academy of Sciences, 110, 19313–19317, 2013) and Benjamin et al. (Nature Human Behaviour, 2, 6–10 2018) recommend using the significance level of α = 0.005 (0.5%) as opposed to the conventional 0.05 (5%) level. Even though their suggestion is easy to implement, it is unclear whether or not the commonly used statistical tests are robust and/or powerful at such a small significance level. Therefore, the main aim of our study is to investigate the robustness and power curve behaviors of independent (unpaired) two-sample tests for metric and ordinal data at nominal significance levels of α = 0.005 and α = 0.05. Through an extensive simulation study, it is found that the permutation versions of the Welch t-test and the Brunner-Munzel test are particularly robust and powerful while the commonly used two-sample tests which utilize t-distribution tend to be either liberal or conservative, and have peculiar power curve behaviors under skewed distributions with variance heterogeneity.

3. https://files.osf.io/v1/resources/ye2d4/providers/osfstorage/60dee6c3f83de20290afec7b?action=download&direct&version=2 for PDF or online viewer: https://osf.io/preprints/psyarxiv/ye2d4

(free full text)

Karch, J. D. (2021). Choosing between the two-sample t test and its alternatives: a practical guideline.. https://doi.org/10.31234/osf.io/ye2d4

It discusses making decisions about appropriate test use. Too wide to paste everything in the guideline. I'm citing it to show the permutation Welch test is actually used and cited.

4. https://www.researchgate.net/publication/24052826_Resampling_Student%27s_t-type_statistics

(free access) Janssen, Arnold. (2005). Resampling Student's t-type statistics. Annals of the Institute of Statistical Mathematics. 57. 507-529. 10.1007/BF02509237.

The present paper establishes conditional and unconditional central limit theorems for various resampling procedures for the t-statistic. The results work under fairly general conditions and the underlying random variables need not to be independent. Specific examples are then the m(n) (double) bootstrap out of k(n) observations, the Bayesian bootstrap and two-sample t-type permutation statistics. In case when m(n)/k(n) is bounded away from zero and infinity necessary and sufficient conditions for the conditional central limit law of the bootstrap t-statistics are established. For high resampling intensity when m(n)/k(n) tends to infinity the following general result is obtained. Without further other assumptions the bootstrap makes the resampled t-statistic automatically normal. The results are based on a general conditional limit theorem for weighted resampling statistics which is of own interest.

5. http://129.217.131.68:8080/bitstream/2003/40978/1/Diss.pdf

Lubna Amro, Resampling-Based Inference Methods for Repeated Measures Data with Missing Values, Dissertation in partial fulfillment of the requirements for the degree of Doktor der Naturwissenschaften submitted to the Department of Statistics TU Dortmund University, 2022

It discusses various aspects of resampling tests. I cite it to show that the permutation Welch t test is discussed also in the recent academic work:

Answer 3

I finally found the paper myself: https://www.sciencedirect.com/science/article/pii/S0167715297000436

According to this paper, the permutation version of the Welch test is preferable over the normal Welch test in many situations.