Skip to main content
changed tag
Link
gung - Reinstate Monica
  • 147.5k
  • 89
  • 406
  • 717
added 608 characters in body; edited tags; edited title
Source Link

Determining sample sizeSample Size for AB test with proportion greater than oneANOVA

So I'm pretty sure I'm thinking about So I'm pretty sure I'm thinking about this all wrong, but.. I'm trying to run an AB test on a web platform that will increase the total number shares to a social network (incredibly exciting, I know). This number has a mean of 30 shares per person, currently. The new mean should be around 40.

How do I determine the sample size for this experiment? Normally we'd throw some numbers into a calculator like baseline mean, the minimal detectable effect, etc. But all these calculators are designed for proportions, i.e. conversion rates, with the assumption that the proportion will be less than zero.

EDIT

We're revisiting this all wrong, but.problem here and it's become much clearer what the goal is. I'm trying to run an AB test on a web platform that will increaseThe outcome of the total number shares toexperiment will affect a social networkcount (incredibly exciting, I know)i. Thise. number has a mean of 30 shares per person, currentlyuser) and there will be multiple experimental variations. The new mean shouldSo the test that we'll end up using to find significance will be around 40ANOVA.

How So the question then becomes, how do I determineyou calculate the sample size/power per experimental variation for this experimentANOVA? Normally we'd throw some numbers into a calculator like baseline mean

For context, the minimal detectable effectvariations A, etc. But all these calculators are designed for proportionsB and C, i.e. conversion rateswhere C is control, with the assumption that the proportion will be less than zerosplit roughly at 5%, 5%, 90% of all traffic respectively. This of course could vary to get enough power faster.

Determining sample size for AB test with proportion greater than one

So I'm pretty sure I'm thinking about this all wrong, but.. I'm trying to run an AB test on a web platform that will increase the total number shares to a social network (incredibly exciting, I know). This number has a mean of 30 shares per person, currently. The new mean should be around 40.

How do I determine the sample size for this experiment? Normally we'd throw some numbers into a calculator like baseline mean, the minimal detectable effect, etc. But all these calculators are designed for proportions, i.e. conversion rates, with the assumption that the proportion will be less than zero.

Determining Sample Size for ANOVA

So I'm pretty sure I'm thinking about this all wrong, but.. I'm trying to run an AB test on a web platform that will increase the total number shares to a social network (incredibly exciting, I know). This number has a mean of 30 shares per person, currently. The new mean should be around 40.

How do I determine the sample size for this experiment? Normally we'd throw some numbers into a calculator like baseline mean, the minimal detectable effect, etc. But all these calculators are designed for proportions, i.e. conversion rates, with the assumption that the proportion will be less than zero.

EDIT

We're revisiting this problem here and it's become much clearer what the goal is. The outcome of the experiment will affect a count (i.e. number of shares per user) and there will be multiple experimental variations. So the test that we'll end up using to find significance will be ANOVA. So the question then becomes, how do you calculate the sample size/power per experimental variation for ANOVA?

For context, the variations A, B and C, where C is control, will be split roughly at 5%, 5%, 90% of all traffic respectively. This of course could vary to get enough power faster.

Source Link

Determining sample size for AB test with proportion greater than one

So I'm pretty sure I'm thinking about this all wrong, but.. I'm trying to run an AB test on a web platform that will increase the total number shares to a social network (incredibly exciting, I know). This number has a mean of 30 shares per person, currently. The new mean should be around 40.

How do I determine the sample size for this experiment? Normally we'd throw some numbers into a calculator like baseline mean, the minimal detectable effect, etc. But all these calculators are designed for proportions, i.e. conversion rates, with the assumption that the proportion will be less than zero.