In many cases the P-value of a 2-sided test is twice the P-value of
a 1-sided test.
For demonstration purposes, here are descriptions of a sample of size $n = 100$ from a Laplace distribution
(centered at 2 and with much heavier tails than normal).
summary(x)
Min. 1st Qu. Median Mean 3rd Qu. Max.
-13.0858 -0.2901 1.0873 0.8850 1.8196 11.6360
sd(x); length(x)
[1] 3.122601
[1] 100
boxplot(x, col="skyblue2", pch = 20, horizontal=T)
Here is a permutation test, in R, of $H_0: \mu = 0$ vs $H_a: \mu \ne 0.$
At each iteration, the sign of each observation is randomly changed between $\pm 1.$ At each iteration, the mean of the sign-permuted data is found.
[That is, the 'metric' of the permutation test is the sample mean.]
The observed mean of the sample is 1.885.
set.seed(1234)
a.obs = mean(x)
a.prm = replicate(10^5, mean(sample(c(-1,1),100,rep=T)*x) )
mean(abs(a.prm) >= abs(a.obs))
[1] 0.00496 # P-value of two-sided test
hist(a.prm, prob=T, col="skyblue2", xlim=c(-7,7))
abline(v=c(a.obs,-a.obs), col="red")
The P-value of the two-sided test is the area in both tails beyond
the vertical red lines.
For a one-sided test (against $H_a: \mu > 0,$ the P-value would be
the value in the right tail beyond 0.885. R-code 'mean(a.prm >= a.obs)`,
which returns 0.00223.
Notes: (1) In the R code for the P-values, the vector with >=
is
a logical vector with $10^5$ TRUE
s and FALSE
s, and the mean is
the proportion of its TRUE
s. (2) Eudey et al. has an elementary presentation of permutation tests; Sect. 2 on paired tests is similar to your one-sample test.
prop.test
. Also, what did you permute, the group labels and then run the same classification model (retrained, I guess) to check for accuracy? $\endgroup$