Website AB testing: Determine time the test can reach statistical significance

Question

THIS IS A UPDATE TO MY ORIGINAL POST:

I have a website A/B test coming up. The test is this.

*I'm testing sale messaging content slot placement on a Home Page

*I have 2 versions of the Home Page created to show different audiences

*I want to be able to determine "by noon" that the results will confirm my lift hypothesis. The information used to confirm this I think would be revenue generated from those who viewed the HP in their visit

Again, On the day a test launches, I want to know if I can determine by noon if it’s successful and statistically significant. I also want to be able to predict ahead of time how long something needs to run to be significant. I'm looking for advice on how to approach this. We use SAS in our office. For Direct mail campains I use a power anysis to calculate sample size but in this situation I’m not sure how to determine sample size and the length of time it will take to reach significance.

A power analysis should work but there are steps in the process im not sure I understand.

*you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost) – I’m not sure what this means

*pick a a desired power threshold (which I would assume would be 0.8) – I understand that

*The Type I error rate for your test (0.05) – I understand that

The power analysis will estimate the sample size needed to meet these conditions. From there, you should be able to use page-view metrics to convert this estimate into time. I’m not sure how to interpret that last statement.

For a DM A/B test I use this code for test and control groups.

proc power;
  TwoSampleFreq
  Test=Fisher
  Alpha = 0.05
  Sides = U  
  GroupProportions = ( /*Historical Benchmark: Response Rate*/ /*lift*/)
  Power =.8
 Npergroup  =.;run;  

Any examples, assistance with this or sample code will be greatly appreciated.

Answer 1

A power analysis should work. In general, you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost), a desired power threshold (0.8 is widely used), and the Type I error rate for your test ($\alpha$). The power analysis will estimate the sample size needed to meet these conditions. From there, you should be able to use page-view metrics to convert this estimate into time. The exact details, of course, will depend on your specific methodology.

An example:

A web traffic report should provide you the number of unique visitors to your site per unit of time, which you could use to estimate the number of individuals who will visit the homepage between the time your test goes live and noon. You could then use this number as the sample size in your power analysis and calculate a minimum detectable effect size (assuming a fixed alpha and power level). This will tell you that by noon, you will either be able to reject your null hypothesis or accept that the difference between options A and B is less than or equal to the effect size. Of course, if the effect size is particularly large, this may not provide you with much substantial information. You could detect a smaller effect size by spending more time running the test (and thus increasing your sample size).

library(pwr)
## Assuming that revenue generated is your response variable
## and that it meets the assumptions of a t-test.  

visitorsPerHour = 15 # More realistically, this number would
                     # vary based on the time of day.
hours = 4 #assuming 8:00 to noon
n = hours*visitorsPerHour # sample size
pwr = pwr.t.test(n = n, sig.level = 0.05, power = 0.8, 
          type = "two.sample", alternative = c("two.sided"))
pwr$d # this is your effect size.
> [1] 0.5157065

0.5 is a pretty large effect size, so you would want to keep testing.

d = 0.1 # Minimum desired effect size    
pwr2 = pwr.t.test(d = d, sig.level = 0.05, power = 0.8, 
           type = "two.sample", alternative = c("two.sided"))
pwr2$n/ visitorsPerHour # number of hours needed.
> [1] 104.7155

Four days of testing may be unacceptably long, and maybe you consider an effect size of 0.1 smaller than necessary. You can play around with different options until you find one that suits your constraints. For reference, the effect size for this test is Cohen's d: $$d=\frac{\bar{x}_{1}-\bar{x}_{2}}{s_{pooled}}$$ If you know the targeted minimum difference in revenue that's worth pursuing (and its standard deviation), you could use that effect size for a more specific answer.

If you run the test forever, of course, you'll eventually get a significant difference even if the size of that difference is practically meaningless.