So I'm pretty sure I'm thinking about this all wrong, but.. I'm trying to run an AB test on a web platform that will increase the total number shares to a social network (incredibly exciting, I know). This number has a mean of 30 shares per person, currently. The new mean should be around 40.
How do I determine the sample size for this experiment? Normally we'd throw some numbers into a calculator like baseline mean, the minimal detectable effect, etc. But all these calculators are designed for proportions, i.e. conversion rates, with the assumption that the proportion will be less than zero.
We're revisiting this problem here and it's become much clearer what the goal is. The outcome of the experiment will affect a count (i.e. number of shares per user) and there will be multiple experimental variations. So the test that we'll end up using to find significance will be ANOVA. So the question then becomes, how do you calculate the sample size/power per experimental variation for ANOVA?
For context, the variations A, B and C, where C is control, will be split roughly at 5%, 5%, 90% of all traffic respectively. This of course could vary to get enough power faster.