What you are doing seems more like a bootstrap procedure than a permutation
test. Because I don't have your data or a description of it, I can't be
sure why you aren't using a Welch 2-sample t test or a 2-sample Wilcoxon rank sum test.
Suppose I have data as below, as sampled in R. Even though sample sizes
are very large, I would not want to trust a t test with such severely
skewed data.
summary(a); length(a); sd(a)
Min. 1st Qu. Median Mean 3rd Qu. Max.
112.2 188.8 285.1 368.7 463.2 1904.2
[1] 2040
[1] 254.4081
summary(b); length(b); sd(b)
Min. 1st Qu. Median Mean 3rd Qu. Max.
56.03 141.93 256.71 347.55 461.82 2611.67
[1] 2741
[1] 292.5565
x = c(a,b); gp = rep(1:2, c(2040,2741))
boxplot(x ~ gp, col="skyblue2", pch=20)
I will not use a t test because I do not trust the t statistic to
have t distribution. However, I will use the pooled 2-sample t test
statistic as my 'metric' for a permutation test, because I feel
the t statistic is a reasonable way to express the difference between
values in groups A and B.
t.obs = t.test(x ~ g, var.eq=T)$stat; t.obs
t
2.613055
Now I will use sample(gp) to randomly permute the 1s and 2s in gp. All $2040 + 2741$ are reassigned to groups 1 and 2 with $n_1 = 2040$ in group 1 and the rest in group 2. On a much smaller scale the code
below illustrates one permutation of the vector c(1,1,1,2,2,2,2).
sample(c(1,1,1,2,2,2,2))
[1] 2 2 2 1 1 2 1
I will do $10\,000$ permutations of gp,
finding t.prm for each permutation. Then the P-value of the permutation test will be the proportion of the values in t.prm that are larger
in absolute value than t.obs for the original unpermuted data.
set.seed(2020)
t.prm = replicate(10^4, t.test(x~sample(gp),var.eq=T)$stat)
mean(abs(t.prm) > abs(t.obs))
[1] 0.0079
The P-value is about $0.008$ so we reject the null hypothesis that the
two groups have the same mean.
hist(t.prm, prob=T, br=30, col="skyblue2",
main="Simulated Permutation Dist'n")
abline(v = c(-t.obs, t.obs), col="red", lwd=2, lty="dotted")
Notes: (1) A Wilcoxon rank sum test shows that the two groups
have significantly different locations:
wilcox.test(x ~ gp)
Wilcoxon rank sum test with continuity correction
data: x by gp
W = 3138800, p-value = 3.737e-13
alternative hypothesis: true location shift is not equal to 0
(2) For a one-sided test, as in your question, the last line of R code for the P-value would be as follows:
mean(t.prm > t.obs)
[1] 0.0042
(3) I am not familiar with the test you did. It seems to be a bootstrap test (because of re-sampling with replacement). I'm not saying your test is wrong; it gives a reasonable result. However, you asked about a permutation test; your test is not a permutation test; so I showed you one.
(4) The R code below was used to obtain the two samples used in
the discussion above:
set.seed(2020)
a = rexp(2040, 1/256) + 112
b = rexp(2741, 1/284) + 56
x = c(a,b); gp = rep(1:2, c(2040,2741)
