2
$\begingroup$

Let's say I test a new version of my signup page. After the original and the variant each get 100 views, here are my results:

  • Original had 1 conversion (1%), and a 95% confidence interval of 0-2.95%
  • Variant had 5 conversions (5%), and a 95% confidence interval of 0.73-9.27% My p-value is 0.047502. So according to the math, I can be 95.2498% sure that the variant is better at some level than the original.

But is it?

Intuitively, I feel like the sample size is too small for 4 conversions to be significant. And I see articles on the internet saying I should wait for some minimum number of conversions. I wonder if it's just the imperfect human in me, or if the traditional statistical calculations are distorted at small sample sizes.

Does anyone know if there is a problem with our formulas with small samples? If so, can you point me to any articles covering the topic?

The calculation I'm using comes from VWO. You can get the same results from their A/B Split Test Significance Calculator and see the formula in their Excel version.

$\endgroup$
  • 6
    $\begingroup$ "So according to the math, I can be 95.2498% sure that the variant is better at some level than the original." -- which mathematics says you can conclude that? I think you misunderstand what a p-value is. $\endgroup$ – Glen_b Apr 7 '15 at 6:20
  • 1
    $\begingroup$ I also can't reproduce your p-value. Are you using a normal approximation to a binomial or something? I think your expectations might be too small for that to work well and you don't seem to be using continuity correction (which should help some with the discreteness). Using a proportions test with continuity correction (prop.test in R) I get p=0.21. With a chi-squared test (Yates correction) I get excatly the same (which I'd expect). With simulation of the null distribution of the chi-square statistic given fixed margins, I get p=0.21. With Fisher's exact test, I get p=0.21. $\endgroup$ – Glen_b Apr 7 '15 at 6:36
  • 3
    $\begingroup$ In short, you should explain exactly which calculations you are performing. $\endgroup$ – Glen_b Apr 7 '15 at 6:37
  • 1
    $\begingroup$ @Glen_b Thanks for pointing out my misunderstanding of a p-value. I just read an article, How to Correctly Interpret P Values that cleared it up. $\endgroup$ – Mike M. Lin Apr 7 '15 at 22:48
  • $\begingroup$ I'm glad it helped, but some of the explanation there is also incorrect. For example, it says "P values evaluate how well the sample data support the [...] argument that the null hypothesis is true" --- p-values really don't do that! Fortunately, later on it gives the correct definition "a P value is the probability of obtaining an effect at least as extreme as the one in your sample data, assuming the truth of the null hypothesis" $\endgroup$ – Glen_b Apr 8 '15 at 2:17
-1
$\begingroup$

Partially answered in comments:

"So according to the math, I can be 95.2498% sure that the variant is better at some level than the original." -- which mathematics says you can conclude that? I think you misunderstand what a p-value is. – Glen_b

I also can't reproduce your p-value. Are you using a normal approximation to a binomial or something? I think your expectations might be too small for that to work well and you don't seem to be using continuity correction (which should help some with the discreteness). Using a proportions test with continuity correction (prop.test in R) I get p=0.21. With a chi-squared test (Yates correction) I get exactly the same (which I'd expect). With simulation of the null distribution of the chi-square statistic given fixed margins, I get p=0.21. With Fisher's exact test, I get p=0.21. – Glen_b

In short, you should explain exactly which calculations you are performing. – Glen_b

( Thanks for pointing out my misunderstanding of a p-value. I just read an article, How to Correctly Interpret P Values that cleared it up. – Mike M. Lin )

I'm glad it helped, but some of the explanation there is also incorrect. For example, it says "P values evaluate how well the sample data support the [...] argument that the null hypothesis is true" --- p-values really don't do that! Fortunately, later on it gives the correct definition "a P value is the probability of obtaining an effect at least as extreme as the one in your sample data, assuming the truth of the null hypothesis" – Glen_b

Summary: You really need a larger sample.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.