THIS IS A UPDATE TO MY ORIGINAL POST:
I have a website A/B test coming up. The test is this.
*I'm testing sale messaging content slot placement on a Home Page
*I have 2 versions of the Home Page created to show different audiences
*I want to be able to determine "by noon" that the results will confirm my lift hypothesis. The information used to confirm this I think would be revenue generated from those who viewed the HP in their visit
Again, On the day a test launches, I want to know if I can determine by noon if it’s successful and statistically significant. I also want to be able to predict ahead of time how long something needs to run to be significant. I'm looking for advice on how to approach this. We use SAS in our office. For Direct mail campains I use a power anysis to calculate sample size but in this situation I’m not sure how to determine sample size and the length of time it will take to reach significance.
A power analysis should work but there are steps in the process im not sure I understand.
*you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost) – I’m not sure what this means
*pick a a desired power threshold (which I would assume would be 0.8) – I understand that
*The Type I error rate for your test (0.05) – I understand that
The power analysis will estimate the sample size needed to meet these conditions. From there, you should be able to use page-view metrics to convert this estimate into time. I’m not sure how to interpret that last statement.
For a DM A/B test I use this code for test and control groups.
proc power; TwoSampleFreq Test=Fisher Alpha = 0.05 Sides = U GroupProportions = ( /*Historical Benchmark: Response Rate*/ /*lift*/) Power =.8 Npergroup =.;run;
Any examples, assistance with this or sample code will be greatly appreciated.