What's the drawbacks of choosing a high statistical power and low significance level? I want to perform an AB test for two products (a control and a variant). I want to detect a 20% uplift with a very high certainty i.e., statistical power and a very low probability of a type I error (significance level). Let's say 99% and 2% respectively.
What is the drawback in doing so — choosing alpha and power to be very low and very high respectively, other than needing a very high sample size for both groups?
 A: You identified the problem: large sample size required. That could get expensive, there could be ethical issues (e.g., human subjects), etc.
A: When you're speaking of power, you must have a
specific alternative value in mind. For example,
if you're testing $H_0: \mu = \mu_0 = 50$ against $H_a: \mu > 50,$ then you need to pick a specific alternative
value, say $\mu_a = 53.$  The power becomes greater as $\Delta = \mu_a - \mu_0$ gets larger.
Suppose $n = 100, \sigma=7, \Delta = 3.$ Then a one-sided, one-sample t test of $H_0:\mu=50$ against
$H_a:\mu > 50$ might reject $H_0$ with a very
small P-value (nearly $0)$ as illustrated below.
set.seed(121)
x = rnorm(100, 53, 7)
t.test(x, mu=50, alt="greater")

        One Sample t-test

data:  x
t = 3.7309, df = 99, p-value = 0.0001592
alternative hypothesis: 
 true mean is greater than 50
95 percent confidence interval:
 51.28982      Inf
sample estimates:
mean of x 
 52.32417

By contrast, if $\Delta = 0.1,$ with everything
else the same, the t test might not reject at
the 5% level, as
illustrated below, where P-value $= 0.0684 > 0.05 = 5\%.$ [In R one can use S-notation to show only the P-value.]
set.seed(2022)
y = rnorm(100, 50.1, 7)
t.test(y, mu=50, alt="greater")$p.val
[1] 0.06837589

Finally, by using a simulation with such t-tests on 100,000 samples from $\mathsf{Norm}(52, 7).$ one can
approximate the power for $n = 100, \sigma=7, \Delta=2),$ and significance level $\alpha=0.05=5\%.$ The answer is about 88% power.
set.seed(2022)
pv = replicate(10^5, t.test(rnorm(100,52,7), 
               mu = 50, alt="greater")$p.val)
mean(pv <= 0.05)
[1] 0.88323

Notes:
(1) If you want high power and low significance level, you may have to use
a large sample size $n$ or a large difference $\Delta$ between hypothetical
and alternative means--or a combination of both.
(2) Many software programs (and some internet
pages) have 'power and sample size' procedures.
Some use simulation and some use an exact formula
based on a non-central t distribution.
