I've read several seemingly conflicting accounts on the best way to calculate sample size. Visual Website Optimizer (VWO) has a lengthy article on this topic. So does Evan Miller. And so does Optimizely.
Using the various tools to estimate sample size with the following settings:
- Baseline Conversion Rate: 3%
- Minimum Detectable Effect: 20%
- Significance: 95%
- Variations: 2
I get the following from the various calculators:
- VWO (have to set "daily visitors" to 1 to get exact sample size): 25,867
- Evan Miller (set to relative, stat. power 80%): 13,050
- Optimizely: 13,000
Given the seemingly different methods of calculation, which one is the "best" to use? I'm trying to understand how to approach this issue of sample size. Thanks!
(I had to list links here because I need more points to post more than 2 inline links) References: