Is "Permutation Test" sufficient for a/b testing? I read the "Practical Statistics for Data Scientists" book which is written by Peter Bruce and Andrew Bruce. In this book, the hypothesis testing is only realized with permutation test. If I searched the web, there are formulas, also I did not see too many permutation test blog articles. I want to ask that the permutation test is enough to analyzing a/b experiment data, or do I need different thing ?
Sorry for my grammatical mistakes, my native language is not English.
 A: Ultimately permutation tests are a type of statistical significance test; we could have use bootstrapping if we want another frequentist non-parametric approach (e.g. see "How do bootstrap and permutation tests work?" (2003) by Janssen & Pauls for a relevant comparison). Whether or not they should be used instead of standard parametric tests is an question that is really old (e.g. see The large-sample power of tests based on permutations of observations (1952) by Hoeffding as an early attempt to use permutation tests instead of standard parametric tests of hypothesis testing) and doesn't really have a definitive answer. It might be even argued that permutation tests construct a null when there is not a well-defined one while parametric test never shy away from the explicit definition of a null - see Chapt. 17 Large-Scale Hypothesis Testing and FDRs in Computer Age Statistical Inference (2016) by Efron & Hastie for a more careful discussion on that.
Permutation tests therefore are not a statistical panacea. As such, think the question should not be if "permutation testing" is sufficient for A/B testing but if "hypothesis testing" is sufficient for A/B testing. In that case, the answer is no. We need regression analysis, we need to be able to be certain that there are not confounding variables that we accidentally ignored or unexpectedly included, and so forth. Yes,  permutation testing is enough in the majority of cases where everything has gone as planned, we observe no seasonality, no primacy or novelty effects, we have limited selection bias, and what not. In order cases, I would strongly urge you to educate yourself in regression analysis and general experimental design principles too. Recently, causal inference has also come to the foreground as a mean to get even greater insights by our observational data but that is a step following standard regression analysis techniques.
Finally, it is notable that while permutation tests are indeed a type of hypothesis testing, A/B testing is a type of an experimentation approach to maximise business goals and using statistical hypothesis testing is only part of the methodology required (parametric or non-parametric, frequentist or Bayesian, etc). For example, multi-armed bandits is another experimentation approach to solve the same problem. Ultimately what we want to maximise our utility functions (e.g. time on client, user expenditure, etc.) That is not what hypothesis testing is aiming to do as it is much more interested in the truthfulness of a hypothesis.
A: Consider the two fictitious normal samples below, in which
sample sizes, sample means, and sample variances all differ.
set.seed(2021)
x1 = rnorm( 70, 50,   5)
x2 = rnorm(100, 53.2, 9)

Someone who did not notice the different variances, might
do a pooled 2-sample t test, find no difference in means
at the 5% level, and conclude nothing interesting is going on.
t.test(x1, x2, var.eq=T)$p.val
[1] 0.07286454

summary(x1); length(x1);  sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  38.72   45.22   49.70   49.55   52.70   60.60 
[1] 70           # sample size
[1] 5.366852     # sample standard deviation

summary(x2); length(x2);  sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  28.89   45.37   51.90   51.87   57.64   77.20 
[1] 100
[1] 9.805152

stripchart(list(x1,x2), ylim=c(.5,2.5), pch="|")


However, a more appropriate Welch 2-sample t test (that does not
assume equal variances) does find a difference in means at the 5% level.
t.test(x1, x2)$p.val
[1] 0.04866419

And (assuming normality) an F-test finds a highly significant difference
between sample variances.
var.test(x1,x2)$p.val
[1] 3.605368e-07

Even the notoriously underpowered Kolmogorov-Smirnov test
finds that the two population distributions are not quite the same.
ks.test(x1,x2)$p.val
[1] 0.04495466

A nonparametric Wilcoxon Rank Sum test finds a significant difference.
However, owing to the different shapes of the samples, this cannot be
considered as a test for difference in medians.
wilcox.test(x1, x2)

        Wilcoxon rank sum test with continuity correction

data:  x1 and x2
W = 2878, p-value = 0.04909
alternative hypothesis: true location shift is not equal to 0

Nor is it clear that the (barely significant) 'location shift' really amounts to stochastic
domination by the second (larger) sample [brown in the empirical
CDF plots below.]
hdr = "ECDFs of Samples 1 [blue] and 2 [brown]"
plot(ecdf(x1), col="blue", main=hdr)
 lines(ecdf(x2), col="brown")


Furthermore, permutation tests using various metrics could be
proposed, which might show a difference in population means---or not.
Depending on what you mean by "better," you might find a standard
or permutation test to support that either A or B is "better"--or that there is not enough "difference" between them to be of practical
importance.
A: It depends. That’s a very broad question.
Permutation testing is one way, of many, for hypothesis testing. In contrast to the more common tests based on sampling distribution, permutation tests tend to be more conservative to accept new evidence (lower power) and less sensitive to assumptions about the population. A downside of these tests is they’re very computationally intensive and impractical for medium or large datasets.
So permutation tests are appropriate for A/B testing if you’re willing to make these trade offs.
