0
$\begingroup$

I need a gut check here...

I'm trying to run an A/B test where the baseline metric is a conversion rate that is close to 1 - somewhere in the 80-85% range.

And so when I run my sample size calculation - i believe i'm just doing a Two Proportion Z-Test - the required sample size is very low, like a few thousand.

This doesn't make much sense to me intuitively? I know that if baseline rate is higher, the sample size will decrease just looking at the formula... but intuitively, I have a hard time understanding how we can gain significance by jut a few thousand samples.

We're working on ads, and a normal CTR ad campaign for us requires ~1M ads for sample size (where CTR is closer to 1%).

Other details: using power = 80%, alpha = 0.05, and an MDE (absolute) of 5% which I suppose is quite high that also contributes?

The reason I chose an MDE of 5% is because stakeholders have said we're ok observing an increase or decrease in the rate by 5% (absolute not relative)

Any thoughts? Should I be using a different calculation?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Every time I have a concern about a power calculation, I simulate it.

The function in R to compute the requisite sample size is power.prop.test.

``` r
p1 = 0.85
p2 = 1.05*0.85

N = ceiling(power.prop.test(p1=p1, p2=p2, power=0.8)$n)

N
#> [1] 974

Created on 2023-03-04 by the reprex package (v2.0.1)

Ok, that seems low. Let's simulate it

sims = replicate(1000, {
  
  x = rbinom(2, N, c(p2, p1))
  test = prop.test(x, rep(N, 2))
  
  (test$p.value<0.05)
})

mean(sims)
> 0.778 # Depending on your random seed

So it looks like this sample size is fine.

Improve this answer
$\endgroup$
5
  • $\begingroup$ thank you for the response. sorry can you clarify what you are doing in your 2nd block of code simulating, and how i should be thinking about your follow up statement? my company has obviously much more daily users to test on, so i feel a bit silly saying "we need 1000 users for this a/b test" which we can achieve in the span of 1 hour.... would i consider changing my MDE and power? is there a separate stat test i should be using? $\endgroup$
    – jc315
    Mar 6 at 1:04
  • $\begingroup$ 2nd block of code is repeatedly sampling two binomial random variables, one with probability of success p2 and the other with probability of success p1. Then, I run a test of proportions and determine how many times the null is rejected. 778 of the 1000 simulations reject the null, hence the power of the test when the sample size in each group is N is approximately 77.8% $\endgroup$
    – Demetri Pananos
    Mar 6 at 4:04
  • $\begingroup$ How should you think about my follow up statement? I had accidentally included a sentence I didn't mean to, so now things should be copacetic. $\endgroup$
    – Demetri Pananos
    Mar 6 at 4:05
  • $\begingroup$ thank you very much. last question if you dont mind since you seem to be well informed on experimentation - how should i approach choosing the appropriate statistical test with A/B testing, aside from whether my metric is a proportion or continuous. for example, do i need to look at the distribution of the data and how would i do that? i apologize, im a fairly new grad and still learning analytics basics $\endgroup$
    – jc315
    Mar 6 at 5:03
  • $\begingroup$ @jc315 that’s not really a question I can answer in the comments. I can say that if the randomization is successful and Reilly independent of all other factors, the type of outcome (e.g. continuous or binary) is all that matters. $\endgroup$
    – Demetri Pananos
    Mar 6 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.