A/B testing-question on ad placement I need help on building the thought process to get to the answer to the following question:
"You want to test which of the two ad placements on your website is better. How many visitors and/or how many times each ad is clicked do we need so that we can be 95% sure that one placement is better?"
I understand it is related to A/B testing, but have no clue how to solve it. Thanks in advance!
 A: Suppose you have counts of clicks for each of
the two placements during a particular period of time.
Then you need to take into consideration the following:
(a) About what proportions of the time do you get clicks on the ad (clicks/visits)? [Proportions near $1/2$ lead to higher variability (thus larger required sample sizes) than do proportions nearer to $0$ or $1.]$
(b) What difference in proportions would you want to be sure to detect? [It will be easier to detect the difference $\Delta = .10$ between $0.70$ and $0.80$ than the difference $\Delta = .01$ between $0.70$ and $0.71.]$
(c) What do you mean by "95% sure"? [Do you want a two-sided test of $H_O: p_1= p_2$ against
$H_1: p_1 \ne p_2$ to be rejected with probability $0.95$ testing at the 5% level of significance when
the true difference is $\Delta$ as in (b)? Thisis called the 'power' of the test.]
There are 'power and sample' size calculators
in various statistical software programs and
also some useful ones online. I know of no simple formula.
Also, you can get good approximations
using simulation. To get you started thinking
about scenarios for your particular situation,
I will show a couple of power approximations
using R and its implementation of prop.test.
n = 1500 # visits to sites with each ad placement
p.1 = .80;  p.2 = .85 # proportions of clicks
set.seed(2021)
pv = replicate(10^5,prop.test(c(rbinom(1,n,p.1),rbinom(1,n,p.2)),
                     c(n,n))$p.val)
mean(pv <.05)
[1] 0.94504  # approx power

The same program with n=2000 gives approximate
power 0.98 and n=1000 gives approximate power 0.83.
The first of the 100,000 iterations in the program rejected $H_0$ with P-value near $0,$ based on simulated observed
proportions about $0.802$ and $0.859,$ respectively.
set.seed(2021)
x.1 = rbinom(1, 1500, .8); x.1
[1] 1203
x.2 = rbinom(1, 1500, .85); x.2
[1] 1289
prop.test(c(x.1,x.2), c(1500,1500))

    2-sample test for equality of proportions 
    with continuity correction

data:  c(x.1, x.2) out of c(1500, 1500)
X-squared = 17.122, df = 1, p-value = 3.506e-05
alternative hypothesis: two.sided
95 percent confidence interval:
  -0.08476269 -0.02990398
sample estimates:
   prop 1    prop 2 
0.8020000 0.8593333

