Permutation tests seem to provide a promising alternative for the unpaired t-test, requiring fewer assumptions. However, the core assumption of the permutation test, exchangeability, implies homogeneity of variances. So, as for the standard Student's t-test, when the variances and the group sizes are not equal, the TypeI error of the permutation test can be inflated. The following R script demonstrates this

typeIerrors <- 0
reps <- 1000
n1 <- 10
sd1 <- 5

n2 <- 30
sd2 <- 1

for (i in 1:reps){

  group1 <- rnorm(n=n1,sd=sd1)
  group2 <- rnorm(n=n2,sd=sd2)
  permRes <- permTS(group1,group2)$p.value
  if (permRes<.05){
    typeIerrors <- typeIerrors+1
cat(sprintf('Type I error rate: %.2f\n',typeIerrors/reps))

Assuming homogeneity of variances seems to be problematic in many cases. The Wikipedia page for the Welch's test (the t-test version, which does not assume homogeneity of variances) notes on this topic: "It is not recommended to pre-test for equal variances and then choose between Student's t-test or Welch's t-test. Rather, Welch's t-test can be applied directly and without any substantial disadvantages to Student's t-test as noted above."

A simple approach to making the permutation test robust to variance heterogeneity might be to use the Welch test statistic as the test statistic. I did some research on this and only found one very short, seemingly unreviewed paper about the topic (http://vetdergikafkas.org/uploads/pdf/pdf_KVFD_779.pdf). Do you know other articles discussing the permutation version of the Welch's t-test? Optimally, discussing why or why not it is a good approach.

I found other, seemingly more complex approaches for the problem of making permutation tests robust towards unequal variances (for example, 1, and 2).

  • $\begingroup$ I've suggested an edit to your post, fixing the broken links to springerlink.com. But, it seems that your two examples (in the last sentence) both point to the same paper. Perhaps you could take a look, whenever possible… $\endgroup$
    – user361019
    Commented Jul 18, 2022 at 4:57

3 Answers 3


There is no need to refer to the Welch's test specifically. The permutation test is an alternative to the general class of two-sample $t$-tests.

In exact tests, like the T-test with equal variance assumption, one must analytically find the closed form of the sampling distribution of the test-statistic under the null hypothesis. For the equal variance T-test, it follows the T-distribution with pooled $n$ degrees of freedom. Welch found an approximate solution to the Fisher-Behren's problem using the Satterthwaite degrees of freedom, and that became the T-test which bears his name.

With the permutation testing, there is no need to make assumption about homoscedasticity or even normality (there are some rather general regularity conditions). You state the hypothesis to be tested:

$$\mathcal{H}_0 : \mu_1 = \mu_2 $$ $$\mathcal{H}_1 : \mu_1 \ne \mu_2 $$

Then randomly permute the group label a large number of times: this distribution is a numerical approximation of the test statistic under the null.

  • 2
    $\begingroup$ As far as I understand, the core statement of this answer: "The permutation test is an alternative to the general class of two-sample 𝑡-tests." is wrong but a widely spread belief. The crux is in the "rather general regularity conditions", which contain exchangeability, which implies homoscedasticity. This leads to inflated Type I errors, when using a standard permutation test to test for mean differences on unequal sample size, unequal variance data. See, for example, academic.oup.com/bioinformatics/article/22/18/2244/317881. $\endgroup$ Commented Apr 29, 2019 at 15:20
  • $\begingroup$ @JulianKarls at a glance, this paper discusses the issues of using the pseudolikelihood, e.g. ignoring dependence structures (and quite severely with $\rho=0.9$). I have some issues with the writing style. Worth passing the question to another SE post where you could supply some code or findings? $\endgroup$
    – AdamO
    Commented Apr 29, 2019 at 16:16
  • $\begingroup$ I added a very small simulation study that shows the alpha inflation. $\endgroup$ Commented Apr 29, 2019 at 17:30
  • $\begingroup$ @JulianKarls Thanks for sharing. Yes that would cause a problem. Why don't you calculate the Welch's t-test test statistic in the permuted datasets? Maybe that's what you're asking in the question. I've always done it that way but I suppose I hadn't thought why. It corrects the $\alpha$ level in your simulation. $\endgroup$
    – AdamO
    Commented Apr 29, 2019 at 20:51

You may be also interested in these papers:

  1. https://projecteuclid.org/journals/annals-of-statistics/volume-41/issue-2/Exact-and-asymptotically-robust-permutation-tests/10.1214/13-AOS1090.full (free access)

page 8/24 enter image description here linking to

JANSSEN , A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the general- ized Behrens–Fisher problem. Statist. Probab. Lett. 36 9–21. MR1491070

https://www.sciencedirect.com/science/article/abs/pii/S0167715297000436 saying that:

It is shown that permutation tests based on studentized statistics are asymptotically exact of size α also under certain extended non-i.i.d. null hypotheses. To demonstrate the principle the results are applied to the generalized two-sample Behrens-Fisher problem for testing equality of the means under general non-parametric heterogeneous error distributions. Within this setting we propose a permutation version of the Welch test which is an extension of Pitman's two-sample permutation test. These results are special cases of a conditional central limit theorem for studentized permutation statistics which also applies to asymptotic power functions.

  1. https://link.springer.com/article/10.3758/s13428-021-01595-5 (free access)

Noguchi, K., Konietschke, F., Marmolejo-Ramos, F. et al. Permutation tests are robust and powerful at 0.5% and 5% significance levels. Behav Res 53, 2712–2724 (2021). https://doi.org/10.3758/s13428-021-01595-5


Recent replication crisis has led to a number of ad hoc suggestions to decrease the chance of making false positive findings. Among them, Johnson (Proceedings of the National Academy of Sciences, 110, 19313–19317, 2013) and Benjamin et al. (Nature Human Behaviour, 2, 6–10 2018) recommend using the significance level of α = 0.005 (0.5%) as opposed to the conventional 0.05 (5%) level. Even though their suggestion is easy to implement, it is unclear whether or not the commonly used statistical tests are robust and/or powerful at such a small significance level. Therefore, the main aim of our study is to investigate the robustness and power curve behaviors of independent (unpaired) two-sample tests for metric and ordinal data at nominal significance levels of α = 0.005 and α = 0.05. Through an extensive simulation study, it is found that the permutation versions of the Welch t-test and the Brunner-Munzel test are particularly robust and powerful while the commonly used two-sample tests which utilize t-distribution tend to be either liberal or conservative, and have peculiar power curve behaviors under skewed distributions with variance heterogeneity.

  1. https://files.osf.io/v1/resources/ye2d4/providers/osfstorage/60dee6c3f83de20290afec7b?action=download&direct&version=2 for PDF or online viewer: https://osf.io/preprints/psyarxiv/ye2d4

(free full text)

Karch, J. D. (2021). Choosing between the two-sample t test and its alternatives: a practical guideline.. https://doi.org/10.31234/osf.io/ye2d4

It discusses making decisions about appropriate test use. Too wide to paste everything in the guideline. I'm citing it to show the permutation Welch test is actually used and cited.

enter image description here

  1. https://www.researchgate.net/publication/24052826_Resampling_Student%27s_t-type_statistics

(free access) Janssen, Arnold. (2005). Resampling Student's t-type statistics. Annals of the Institute of Statistical Mathematics. 57. 507-529. 10.1007/BF02509237.

The present paper establishes conditional and unconditional central limit theorems for various resampling procedures for the t-statistic. The results work under fairly general conditions and the underlying random variables need not to be independent. Specific examples are then the m(n) (double) bootstrap out of k(n) observations, the Bayesian bootstrap and two-sample t-type permutation statistics. In case when m(n)/k(n) is bounded away from zero and infinity necessary and sufficient conditions for the conditional central limit law of the bootstrap t-statistics are established. For high resampling intensity when m(n)/k(n) tends to infinity the following general result is obtained. Without further other assumptions the bootstrap makes the resampled t-statistic automatically normal. The results are based on a general conditional limit theorem for weighted resampling statistics which is of own interest.

enter image description here


Lubna Amro, Resampling-Based Inference Methods for Repeated Measures Data with Missing Values, Dissertation in partial fulfillment of the requirements for the degree of Doktor der Naturwissenschaften submitted to the Department of Statistics TU Dortmund University, 2022

It discusses various aspects of resampling tests. I cite it to show that the permutation Welch t test is discussed also in the recent academic work:

enter image description here

  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Commented Dec 24, 2023 at 14:41
  • 1
    $\begingroup$ I updated the post. Thank you for guiding me on how to make the response better. Also, if possible, please withdraw the "-1" downvote, if you find the changes appropriate. $\endgroup$ Commented Dec 27, 2023 at 1:12

I finally found the paper myself: https://www.sciencedirect.com/science/article/pii/S0167715297000436

According to this paper, the permutation version of the Welch test is preferable over the normal Welch test in many situations.

  • 1
    $\begingroup$ Could you add more details? I cannot access the paper. $\endgroup$
    – user271536
    Commented Jan 6, 2021 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.