1
$\begingroup$

I have a conceptual question to ask that may come off dumb, but I'm just trying to see if I understand A/B testing properly as I've only studied it but not worked much in practice.

Let's say we are running an A/B test on ads on some mobile app on a homepage. This experiment tests some UI change, and we are concerned with CTR

We calculate the sample size required based off CTR baseline rates, MDE = 1%, alpha = 0.05, and power = 0.8

Let's say we get a sample size required of 1M, and our daily active users (DAU) is 10M. So we conclude we need to allocate ~10% of users to this experiment, and we choose to run it for 28 days because we want to test a 21-day retention metric as well

Now our team says because of constraints, we can only run this experiment for 10 days.

  1. Does this impact sample size calculations?
  2. Does the % of DAU we need to allocate change?

Intuitively, #2 seems like a yes - meaning if we are going to run the test for a shorter length we would want to increase sample size to get the same confidence in our result.

But looking at my inputs for calculating these numbers, the answer seems no? i.e. the sample size will still be 1M, and we still get 10M DAU?

Can someone please point out where I might have faulty logic/poor understanding of A/B testing here?

Improve this question
$\endgroup$
5
  • $\begingroup$ Retention is not included in the sample size calculation because it's not part of the test, retention is something which you decided to add as a bonus, and you decided to measure it for 21 days. For ex., if you were only interested in the retention for the next 7 days then you would measure only for 7 days. But the test tells you that you need 1M users to get an estimate of CTR. $\endgroup$
    – user2974951
    May 24 at 6:42
  • $\begingroup$ so specifically for CTR - does the % we are allocating for our experiment change if we are shrinking the duration of experiment time? to me our sample size calculation gave us 1M and that does not change, and our DAU does not change, so i dont see why the % of users would change? $\endgroup$
    – jc315
    May 24 at 7:00
  • $\begingroup$ CTR doesn't change if you change the duration of the experiment. As I've mentioned, duration is not included in the estimation of sample size, it is something which you decide to include or not, if you want to test retention as well. $\endgroup$
    – user2974951
    May 24 at 8:34
  • $\begingroup$ so there is no impact on what % of users we must allocate to our experiment, if we only have 5 weeks to run vs 1 week to run, if we are only measuring CTR? assuming we have 10x DAU compared to our required sample size? $\endgroup$
    – jc315
    May 24 at 14:16
  • $\begingroup$ What is retention and how do you want to test it (power, significance, etc)? What is MDE? What baseline rates are you assuming? $\endgroup$
    – Sextus Empiricus
    May 24 at 14:42

1 Answer 1

Reset to default
0
$\begingroup$

a rule of thumb formula for minimum sample size is: $n = \frac{16 * \sigma^2}{\delta^2}$, there are two things that need more clarify:

  • n is the minimum sample size for each group: if you have an unequal sample size for treat/control group, you can refer here(R - power.prop.test, prop.test, and unequal sample sizes in A/B tests)[https://stats.stackexchange.com/questions/108226/r-power-prop-test-prop-test-and-unequal-sample-sizes-in-a-b-tests]

  • n is 「unique active users during experiment」not 「daily active users (DAU) * duration days」 for each group

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.