Safely determining sample size for A/B testing I am a software engineer looking to build an A/B testing tool. I don't have a solid stats background but have been doing quite a bit of reading over the last few days.
I am following the methodology described here and will summarize the relevant points, below.
The tool will allow designers and domain experts to configure a website to split traffic received at a specific URL between two or more URLs. For example, traffic arriving at http://example.com/hello1 could be split between http://example.com/hello1 and http://example.com/hello2. Traffic would be split evenly between target URLs and the performance of the marketing processes at each of the target URLs will be compared.
In this experiment, the sample size N will correspond to visitors. The test will measure "conversions", a term describing when a visitor commits to a specific action in a marketing process. Conversions are expressed in percentages and a higher conversion rate is desirable. This makes the test a comparison of independent proportions. The tool needs to be able to be easily employed to produce tests with safe results. Selecting an appropriate value of N is important.
In the linked article, above, a power analysis of two independent proportions is employed to find N. This method requires that one know the conversion rate of the control in advance as well as specify the target desired conversion improvement. It also specifies a significance level of 95% and a statistical power of 80%.
Questions:


*

*Is this method of determining N sound? If so, what is the safest way to determine the conversion rate of the control prior to beginning the test?

*Are there sound ways of determining N that don't require that one know conversion rates of the control in advance?

*Is the methodology in the linked article sound? If not, are there any accessible and easily digestible methods out there that you could link me to?

 A: IMHO, as far as it goes, the post goes into the right direction. However:


*

*The proposed method implicitly makes two assumptions: the baseline conversion rate and the expected amount of change. The sample size depends very much on how good you meet these assumptions. I recommend that you calculate required sample sizes for several combinations of p1 and p2 that you think are realistic. That will give you a feeling about how reliable the sample size calculation actually is.  
> power.prop.test (p1=0.1, p2 = 0.1*1.1, sig.level=0.05, power=0.8)

     Two-sample comparison of proportions power calculation 

              n = 14750.79
             p1 = 0.1
             p2 = 0.11
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

 NOTE: n is number in *each* group 

> power.prop.test (p1=0.09, p2 = 0.09*1.1, sig.level=0.05, power=0.8)

     Two-sample comparison of proportions power calculation 

              n = 16582.2
             p1 = 0.09
             p2 = 0.099
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

 NOTE: n is number in *each* group 

So if the actual conversion rate is 9% instead of 10%, you need another 2000 cases for each scenario to detect the 10%-more-than-baseline conversion rate of the new form.
After the test is done, you can calculate confidence intervals for the proportions based on your actual observations. 


*

*the last conclusion under 3. (about testing multiple scenarios) is not quite correct. To adjust for multiple testing (in the example multiple = 2), it is not enough to add just another $n$ tests for each new scenario:
If neither B nor C are better than the original version A, and the two tests A ./. B and B ./. C are done as proposed there with $n$ cases for each of the scenarios, then the probability to falsely change away from A is (1 - α)² ≈ 10% (α: accepted probability of type I error; sig.level above). In other words, it is almost twice as large as specified initially. The second problem with that approach is: can you really do without comparing B ./. C? What are you going to do if you find both B and C better than A?

A: The most common method for doing this kind of testing is with binomial proportion confidence intervals (see http://bit.ly/fa2K7B$^\dagger$)
You won't be able to ever know the "true" conversion rate of the two paths, but this will give you the ability to say something to the effect "With 99% confidence, A is more effective at converting than B".
For example:  Lets assume that you have run 1000 trials down path A.  Of these 1000 trials, 121 were successful conversions (conversion rate of 0.121) and we would like a 99% confidence interval around this 0.121 result.  The z-score for 99% confidence intervals is 2.576 (you just look this up in a table), so according to the formula:
$$
\begin{aligned}
\hat p &\pm 2.576\left(\sqrt{\frac{0.121 \times (1 - 0.121)}{1000}}\right)  \\
\hat p &\pm 0.027
\end{aligned}
$$
So with 99% confidence we can say that $0.094 \le \hat p \le 0.148$, where $\hat p$ is the "true" conversion rate of process A.
If we construct a similar interval for process B, we can compare the intervals.  If the intervals don't overlap, then we can say with 98% confidence that one is better than the other. (Remember, we're only 99% confident about each interval, so our overall confidence about the comparison is 0.99 * 0.99)
If the intervals do overlap, then we have to run more trials, or decide that they are too similar in performance to distinguish, which brings us the tricky part - determining $N$, the number of trials.  I'm not familiar with other methods, but with this method, you aren't going to be able to determine $N$ up front unless you have an accurate estimate of the performance of both A and B up front.  Otherwise, you are just going to have to run trials until you get samples so that the intervals separate.
Best of luck to you. (I'm rooting for process B, by the way).

$^\dagger$ The link doesn't work.
A: Instead of calculating overlapping intervals you calculate the Z-score. This is algorithmically easier to implement, and you will get statistical libraries to help.
Take a look here.
