1
$\begingroup$

THIS IS A UPDATE TO MY ORIGINAL POST:

I have a website A/B test coming up. The test is this.

*I'm testing sale messaging content slot placement on a Home Page

*I have 2 versions of the Home Page created to show different audiences

*I want to be able to determine "by noon" that the results will confirm my lift hypothesis. The information used to confirm this I think would be revenue generated from those who viewed the HP in their visit

Again, On the day a test launches, I want to know if I can determine by noon if it’s successful and statistically significant. I also want to be able to predict ahead of time how long something needs to run to be significant. I'm looking for advice on how to approach this. We use SAS in our office. For Direct mail campains I use a power anysis to calculate sample size but in this situation I’m not sure how to determine sample size and the length of time it will take to reach significance.

A power analysis should work but there are steps in the process im not sure I understand.

*you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost) – I’m not sure what this means

*pick a a desired power threshold (which I would assume would be 0.8) – I understand that

*The Type I error rate for your test (0.05) – I understand that

The power analysis will estimate the sample size needed to meet these conditions. From there, you should be able to use page-view metrics to convert this estimate into time. I’m not sure how to interpret that last statement.

For a DM A/B test I use this code for test and control groups.

proc power;
  TwoSampleFreq
  Test=Fisher
  Alpha = 0.05
  Sides = U  
  GroupProportions = ( /*Historical Benchmark: Response Rate*/ /*lift*/)
  Power =.8
 Npergroup  =.;run;  

Any examples, assistance with this or sample code will be greatly appreciated.

$\endgroup$
2
  • $\begingroup$ Welcome to CV. You need to provide more details about what it is you're testing, how you're testing it and what information you think will enable you to make a determination "by noon" that the results will confirm your hypothesis. Without that, people reading your question won't have much to go on to facilitate a response. $\endgroup$ Nov 13, 2015 at 22:28
  • 1
    $\begingroup$ Try this link out: dataorigami.net/collections/bayesian-methods/products/… The guy who runs this site had an interesting talk about AB testing on Data Skeptic podcast. $\endgroup$
    – conv3d
    Nov 14, 2015 at 5:46

1 Answer 1

1
$\begingroup$

A power analysis should work. In general, you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost), a desired power threshold (0.8 is widely used), and the Type I error rate for your test ($\alpha$). The power analysis will estimate the sample size needed to meet these conditions. From there, you should be able to use page-view metrics to convert this estimate into time. The exact details, of course, will depend on your specific methodology.

An example:

A web traffic report should provide you the number of unique visitors to your site per unit of time, which you could use to estimate the number of individuals who will visit the homepage between the time your test goes live and noon. You could then use this number as the sample size in your power analysis and calculate a minimum detectable effect size (assuming a fixed alpha and power level). This will tell you that by noon, you will either be able to reject your null hypothesis or accept that the difference between options A and B is less than or equal to the effect size. Of course, if the effect size is particularly large, this may not provide you with much substantial information. You could detect a smaller effect size by spending more time running the test (and thus increasing your sample size).

library(pwr)
## Assuming that revenue generated is your response variable
## and that it meets the assumptions of a t-test.  

visitorsPerHour = 15 # More realistically, this number would
                     # vary based on the time of day.
hours = 4 #assuming 8:00 to noon
n = hours*visitorsPerHour # sample size
pwr = pwr.t.test(n = n, sig.level = 0.05, power = 0.8, 
          type = "two.sample", alternative = c("two.sided"))
pwr$d # this is your effect size.
> [1] 0.5157065

0.5 is a pretty large effect size, so you would want to keep testing.

d = 0.1 # Minimum desired effect size    
pwr2 = pwr.t.test(d = d, sig.level = 0.05, power = 0.8, 
           type = "two.sample", alternative = c("two.sided"))
pwr2$n/ visitorsPerHour # number of hours needed.
> [1] 104.7155

Four days of testing may be unacceptably long, and maybe you consider an effect size of 0.1 smaller than necessary. You can play around with different options until you find one that suits your constraints. For reference, the effect size for this test is Cohen's d: $$d=\frac{\bar{x}_{1}-\bar{x}_{2}}{s_{pooled}}$$ If you know the targeted minimum difference in revenue that's worth pursuing (and its standard deviation), you could use that effect size for a more specific answer.

If you run the test forever, of course, you'll eventually get a significant difference even if the size of that difference is practically meaningless.

$\endgroup$
3
  • $\begingroup$ If you could elaborate on "From there, you should be able to use page-view metrics to convert this estimate into time." that would be there helpful. I'm not sure how I would use page view metrics to to convert the power analysis results into time. Thanks. $\endgroup$ Nov 18, 2015 at 15:04
  • $\begingroup$ I've updated my response. Does that clear things up? I could provide R code, but I haven't used SAS in years. $\endgroup$ Nov 18, 2015 at 20:37
  • $\begingroup$ Does that help? $\endgroup$ Nov 18, 2015 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.