How does violation of normality conditions affect p-values (relative to permutation tests) I am running some t-tests using data from small samples (n=25) whose distributions do not appear to be normally distributed (they are not extremely skewed, but the skew is apparent in the plots and they fail the Shapiro-Wilks test for normality).
When I run a permutation test (to test for the differences being zero), I noticed the p-values are all higher than when I run the 2 sample t-tests. Is this always true? And if so, why?
Thanks,
Matt Brenneman
 A: Here are some fictitious data for illustration:
set.seed(2022)
x1 = rgamma(25, 2, 1/2)
x2 = rgamma(25, 2, 1/5)

Boxplots show some skewness and seem to show different locations.
boxplot(x1, x2, horizontal=T, col="skyblue2", pch=20)


Also, a Shapiro-Wilk test finds the first sample to be non-normal.
shapiro.test(x1)$p.val
[1] 0.02834856
shapiro.test(x2)$p.val
[1] 0.07122469

I would not choose a t test to look for significantly different locations--especially not a pooled t test, which assumes equal variances.
But it does find a highly significant difference
between the two samples.
t.test(x1, x2, var.eq=T)

        Two Sample t-test

data:  x1 and x2
t = -4.0511, df = 48, p-value = 0.0001854
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -7.345875 -2.472699
sample estimates:
mean of x mean of y 
 4.396683  9.305970 

If we doubt that the t statistic has anything like
Student's t distribution with DF = 48, we might be wary of quoting the P-value above as meaningful.
However, we can use the pooled t-test as a 'metric'; that is, a measure of different population means based on the two samples.
Then we can scramble the 50 observations
repeatedly into two 'samples' of 25,
and find the pooled t statistic for each
scrambling. That will give us an idea of
the actual permutation distribution of
the pooled t statistic. [In R sample(g) does
the scrambling.]
set.seed(1234)
x = c(x1,x2);  g = rep(1:2, each=25)
t = replicate(10^5, t.test(x~sample(g), var.eq=T)$stat)
mean(abs(t) >= 4.0511)
[1] 0.00012  # aprx P-val of permutation test

The P-value of the permutation test is near $0$
so the the two sample means are clearly significantly different. (But this P-value is
not as small as the P-value from the pooled t test--of which
we were suspicious.)
A histogram of the simulated t values is shown
below along with vertical lines at the t statistic
from the pooled t test (and its negative for a 2-sided test), and the density function
of $\mathsf{T}(\nu=48),$ for comparison.
[Agreement in the central part of the distributions is good, but we have seen that the tails are not the same.]
hdr = "Simulated Permutation Dist'n"
hist(t, prob=T, col="skyblue2", main=hdr)
 abline(v = c(-4.05, 4.05), lwd=2, lty="dotted",col="darkgreen") 
 curve(dt(x,48), add=T, lwd=2, col="orange")


Note: @whuber's question in Comments is crucial. You do not say what metric you used for your permutation test.
Different metrics will give different P-values,
so I can't say whether your results are correct. For my fictitious data and my metric the P-value of my permutation test is also larger than
the P-value of the pooled t test.
