1
$\begingroup$

So I'm pretty sure I'm thinking about this all wrong, but.. I'm trying to run an AB test on a web platform that will increase the total number shares to a social network (incredibly exciting, I know). This number has a mean of 30 shares per person, currently. The new mean should be around 40.

How do I determine the sample size for this experiment? Normally we'd throw some numbers into a calculator like baseline mean, the minimal detectable effect, etc. But all these calculators are designed for proportions, i.e. conversion rates, with the assumption that the proportion will be less than zero.

EDIT

We're revisiting this problem here and it's become much clearer what the goal is. The outcome of the experiment will affect a count (i.e. number of shares per user) and there will be multiple experimental variations. So the test that we'll end up using to find significance will be ANOVA. So the question then becomes, how do you calculate the sample size/power per experimental variation for ANOVA?

For context, the variations A, B and C, where C is control, will be split roughly at 5%, 5%, 90% of all traffic respectively. This of course could vary to get enough power faster.

Improve this question
$\endgroup$
7
  • $\begingroup$ Do you know the standard deviation? Do you expect the treatment to alter it? $\endgroup$
    – dimitriy
    Commented Dec 18, 2014 at 2:36
  • $\begingroup$ I'm not expecting to see a change in the standard deviation between the experiment and control. Unfortunately I can't get to an actual number for the SD right now since the data is hard to get to. Is it completely necessary to have? $\endgroup$
    – paulsef11
    Commented Dec 18, 2014 at 6:11
  • $\begingroup$ You will need the sd eventually to sure, but I can repeat the calculation for range. Can the treatment "harm"? $\endgroup$
    – dimitriy
    Commented Dec 18, 2014 at 6:22
  • $\begingroup$ If by "harm" you mean decrease the mean, then yes, it's possible. Can you share with me the process that you're using? $\endgroup$
    – paulsef11
    Commented Dec 18, 2014 at 6:27
  • $\begingroup$ It looks you're dealing with counts (0,1,2,...), not proportions. Your title is misleading/confusing. If you're prepared to assume a Poisson or Negative Binomial model it may be possible to do some kind of similar calculation. $\endgroup$
    – Glen_b
    Commented Dec 19, 2014 at 4:46

1 Answer 1

Reset to default
1
$\begingroup$

Here are the sample size calculations for a two-sample means test. This assumes an equal split between T (group 2) and C (group 1), which maximizes power. The diff column is the minimum detectable difference given alpha (the significance level), the power, and the standard deviation. N is the total sample size. 

. power twomeans 30, sd(5(5)30) alpha(.05 .1) power(.8 .9) diff(-10 10) 

Performing iteration ...

Estimated sample sizes for a two-sample means test
t test assuming sd1 = sd2 = sd
Ho: m2 = m1  versus  Ha: m2 != m1

  +---------------------------------------------------------------------------------+
  |   alpha   power       N      N1      N2   delta      m1      m2    diff      sd |
  |---------------------------------------------------------------------------------|
  |     .05      .8      12       6       6     -10      30      20     -10       5 |
  |     .05      .8      34      17      17     -10      30      20     -10      10 |
  |     .05      .8      74      37      37     -10      30      20     -10      15 |
  |     .05      .8     128      64      64     -10      30      20     -10      20 |
  |     .05      .8     200     100     100     -10      30      20     -10      25 |
  |     .05      .8     286     143     143     -10      30      20     -10      30 |
  |     .05      .8      12       6       6      10      30      40      10       5 |
  |     .05      .8      34      17      17      10      30      40      10      10 |
  |     .05      .8      74      37      37      10      30      40      10      15 |
  |     .05      .8     128      64      64      10      30      40      10      20 |
  |     .05      .8     200     100     100      10      30      40      10      25 |
  |     .05      .8     286     143     143      10      30      40      10      30 |
  |     .05      .9      14       7       7     -10      30      20     -10       5 |
  |     .05      .9      46      23      23     -10      30      20     -10      10 |
  |     .05      .9      98      49      49     -10      30      20     -10      15 |
  |     .05      .9     172      86      86     -10      30      20     -10      20 |
  |     .05      .9     266     133     133     -10      30      20     -10      25 |
  |     .05      .9     382     191     191     -10      30      20     -10      30 |
  |     .05      .9      14       7       7      10      30      40      10       5 |
  |     .05      .9      46      23      23      10      30      40      10      10 |
  |     .05      .9      98      49      49      10      30      40      10      15 |
  |     .05      .9     172      86      86      10      30      40      10      20 |
  |     .05      .9     266     133     133      10      30      40      10      25 |
  |     .05      .9     382     191     191      10      30      40      10      30 |
  |      .1      .8       8       4       4     -10      30      20     -10       5 |
  |      .1      .8      28      14      14     -10      30      20     -10      10 |
  |      .1      .8      58      29      29     -10      30      20     -10      15 |
  |      .1      .8     102      51      51     -10      30      20     -10      20 |
  |      .1      .8     156      78      78     -10      30      20     -10      25 |
  |      .1      .8     224     112     112     -10      30      20     -10      30 |
  |      .1      .8       8       4       4      10      30      40      10       5 |
  |      .1      .8      28      14      14      10      30      40      10      10 |
  |      .1      .8      58      29      29      10      30      40      10      15 |
  |      .1      .8     102      51      51      10      30      40      10      20 |
  |      .1      .8     156      78      78      10      30      40      10      25 |
  |      .1      .8     224     112     112      10      30      40      10      30 |
  |      .1      .9      12       6       6     -10      30      20     -10       5 |
  |      .1      .9      36      18      18     -10      30      20     -10      10 |
  |      .1      .9      80      40      40     -10      30      20     -10      15 |
  |      .1      .9     140      70      70     -10      30      20     -10      20 |
  |      .1      .9     216     108     108     -10      30      20     -10      25 |
  |      .1      .9     310     155     155     -10      30      20     -10      30 |
  |      .1      .9      12       6       6      10      30      40      10       5 |
  |      .1      .9      36      18      18      10      30      40      10      10 |
  |      .1      .9      80      40      40      10      30      40      10      15 |
  |      .1      .9     140      70      70      10      30      40      10      20 |
  |      .1      .9     216     108     108      10      30      40      10      25 |
  |      .1      .9     310     155     155      10      30      40      10      30 |
  +---------------------------------------------------------------------------------+

The formulas used by Stata can be found here.

A free software alternative that easily handles unequal T-C allocations is G*Power.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The trouble here is that we don't want to assume a 50-50 split (i.e. the sample sizes shouldn't be the same). Looking through the STATA docs, there does seem to be support for that but it doesn't go into as much detail... $\endgroup$
    – paulsef11
    Commented Dec 18, 2014 at 17:35
  • $\begingroup$ Sample sizes and the experimental-group means are obtained by iteratively solving the nonlinear equation (5). There's no easy formula. If you modify the original question with what sort of allocation ratio you're considering, I can modify the answer. $\endgroup$
    – dimitriy
    Commented Dec 18, 2014 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.