# Mistake in statistical significance calculator?

In this example, the confidence level is around 95% (P-Value=0.957...) for a difference between the 2 conversion rates of 17.91% (ABS(5.5%-6.7%)/6.7%).

I found another statistical significance calculator from the world's most widely used A/B testing solution which tells me that for the same base conversion rate (6.7%), the same minimum detectable effect (17.91%), and same statistical significance (95%), my sample size should be 2x what I actually have (6.2K vs. 3K)

Which brings me to my question:

Which of these calculators (if any) is actually correct?

• A z score of about 172 should yield an absurdly low p-value (way beyond machine precision). So there must be something wrong with the spreadsheet at the very least. Also a p-value of ~ .96 isn't significant at any of those levels. Hopefully no one actually uses that spreadsheet. – einar Dec 1 '17 at 14:16
• Confidence level is not a synonym for P-value. – Nick Cox Dec 1 '17 at 14:21
• @einar: ok it seems the z score from the spreadsheet is correct by a factor of 100, as I tested other tools (medium.com/@rikhigham/…, github.com/jamesspittal/ab-testing-calculator) and they provide the same z score (by a factor of 100). – Max Dec 1 '17 at 16:12
• Do you really want to be using a tool that was created by someone who obviously is unfamiliar with what it is intended to do, especially when there are thousands of alternatives to choose from? Observing that one of its most basic outputs might consistently be 100 times greater than it should be gives little comfort! – whuber Dec 1 '17 at 19:20
• @whuber: I ended up going for experimentationhub.com/p-value.html – Max Dec 1 '17 at 19:29