Let's say I test a new version of my signup page. After the original and the variant each get 100 views, here are my results:
- Original had 1 conversion (1%), and a 95% confidence interval of 0-2.95%
- Variant had 5 conversions (5%), and a 95% confidence interval of 0.73-9.27% My p-value is 0.047502. So according to the math, I can be 95.2498% sure that the variant is better at some level than the original.
But is it?
Intuitively, I feel like the sample size is too small for 4 conversions to be significant. And I see articles on the internet saying I should wait for some minimum number of conversions. I wonder if it's just the imperfect human in me, or if the traditional statistical calculations are distorted at small sample sizes.
Does anyone know if there is a problem with our formulas with small samples? If so, can you point me to any articles covering the topic?