I need a gut check here...
I'm trying to run an A/B test where the baseline metric is a conversion rate that is close to 1 - somewhere in the 80-85% range.
And so when I run my sample size calculation - i believe i'm just doing a Two Proportion Z-Test - the required sample size is very low, like a few thousand.
This doesn't make much sense to me intuitively? I know that if baseline rate is higher, the sample size will decrease just looking at the formula... but intuitively, I have a hard time understanding how we can gain significance by jut a few thousand samples.
We're working on ads, and a normal CTR ad campaign for us requires ~1M ads for sample size (where CTR is closer to 1%).
Other details: using power = 80%, alpha = 0.05, and an MDE (absolute) of 5% which I suppose is quite high that also contributes?
The reason I chose an MDE of 5% is because stakeholders have said we're ok observing an increase or decrease in the rate by 5% (absolute not relative)
Any thoughts? Should I be using a different calculation?