We have a developed an alternative website which we believe will increase sales by a small percentage (1-4%). Although small this is very valuable.

Our Site:

Visitors are tracked by a cookie and take up to ten days to complete a purchase. Traffic and sales are clearly seasonal.

Despite this we are reasonably comfortable using the A/B testing described in detail on the web and here on Cross Validated.

To determine power, we are performing the following simulation:

  1. Estimate current site conversion using the last ten days of traffic and orders.
  2. Assume that the new site will increase conversion by 2%
  3. Simulate traffic using a binomial random generator. One group will have a probability of success equal to the measured conversion X, the other X * 1.02
  4. Count the number of successes in each group and then run a pearson chi square test on the two proportions.
  5. Repeat this 10,000 times.
  6. Calculate what % of the 10,000 simulations give a significant result (p < 0.05).

This gives a far more conservative sample size to just using power.prop.test in R (say).

Is this a reasonable way to estimate power. Are we are being too conservative? It seems to me that power.prop.test is assuming that 'base' conversion is known, which is not the case in our test.

R snippet:

prop.test.power_analysis = function(runs, confidence_level, trials, successes) { 

 alpha = 1 - confidence_level

 result <- replicate(
 n = runs,
 expr = {

   res <- prop.test(
      c(sum(rbinom(n = trials[1],size = 1,prob = successes[1]/trials[1]))
       ,sum(rbinom(n = trials[2],size = 1,prob = successes[2]/trials[2]))),
     n = trials, alternative = 'greater')$p.value < alpha


power <- sum(result)/runs # power

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.