I have a conceptual question to ask that may come off dumb, but I'm just trying to see if I understand A/B testing properly as I've only studied it but not worked much in practice.
Let's say we are running an A/B test on ads on some mobile app on a homepage. This experiment tests some UI change, and we are concerned with CTR
We calculate the sample size required based off CTR baseline rates, MDE = 1%, alpha = 0.05, and power = 0.8
Let's say we get a sample size required of 1M, and our daily active users (DAU) is 10M. So we conclude we need to allocate ~10% of users to this experiment, and we choose to run it for 28 days because we want to test a 21-day retention metric as well
Now our team says because of constraints, we can only run this experiment for 10 days.
- Does this impact sample size calculations?
- Does the % of DAU we need to allocate change?
Intuitively, #2 seems like a yes - meaning if we are going to run the test for a shorter length we would want to increase sample size to get the same confidence in our result.
But looking at my inputs for calculating these numbers, the answer seems no? i.e. the sample size will still be 1M, and we still get 10M DAU?
Can someone please point out where I might have faulty logic/poor understanding of A/B testing here?