Looking on Quora, I have found a spreadsheet template which calculates statistical significance:

enter image description here

In this example, the confidence level is around 95% (P-Value=0.957...) for a difference between the 2 conversion rates of 17.91% (ABS(5.5%-6.7%)/6.7%).

I found another statistical significance calculator from the world's most widely used A/B testing solution which tells me that for the same base conversion rate (6.7%), the same minimum detectable effect (17.91%), and same statistical significance (95%), my sample size should be 2x what I actually have (6.2K vs. 3K)

enter image description here

Which brings me to my question:

Which of these calculators (if any) is actually correct?

  • 2
    $\begingroup$ A z score of about 172 should yield an absurdly low p-value (way beyond machine precision). So there must be something wrong with the spreadsheet at the very least. Also a p-value of ~ .96 isn't significant at any of those levels. Hopefully no one actually uses that spreadsheet. $\endgroup$ – einar Dec 1 '17 at 14:16
  • 2
    $\begingroup$ Confidence level is not a synonym for P-value. $\endgroup$ – Nick Cox Dec 1 '17 at 14:21
  • $\begingroup$ @einar: ok it seems the z score from the spreadsheet is correct by a factor of 100, as I tested other tools (medium.com/@rikhigham/…, github.com/jamesspittal/ab-testing-calculator) and they provide the same z score (by a factor of 100). $\endgroup$ – Max Dec 1 '17 at 16:12
  • 1
    $\begingroup$ Do you really want to be using a tool that was created by someone who obviously is unfamiliar with what it is intended to do, especially when there are thousands of alternatives to choose from? Observing that one of its most basic outputs might consistently be 100 times greater than it should be gives little comfort! $\endgroup$ – whuber Dec 1 '17 at 19:20
  • $\begingroup$ @whuber: I ended up going for experimentationhub.com/p-value.html $\endgroup$ – Max Dec 1 '17 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.