I am running an A/B test for our site. And get below stats:

Group A: Visitors: 155,590; Conversion: 18,529

Group B: Visitors: 155,592; Conversion: 18,118

And I am using the tool on this website to determine if the test result is significant or not. https://abtestguide.com/calc/

The P-value is 0.9889 which is way higher than 0.05, but the tool still says it's significant. I am very confused. I thought only if the p-value is smaller than 0.05, I can say it's significant (assuming 95% confidence).

  • 1
    $\begingroup$ What tool on what website? $\endgroup$
    – Peter Flom
    Nov 19 '19 at 13:07
  • $\begingroup$ @PeterFlom-ReinstateMonica sorry forgot the link. Just added. $\endgroup$ Nov 20 '19 at 1:52

I have entered your numbers into the form and the result is shown to be significant for the two-sided test but not for the one-sided test whilst the p-value is constant. If you change A and B values both tests (one- and two-tailed) are significant and the p-value becomes < 0.05.

The p-value shown seems to be that of a one-sided test with the order in which you give the data determines the direction of that test, even if you perform a two-sided tests. Two-sided tests are usually considered standard and I consider it bad practice, to show one-sided p-values when a two-sided test is ticked.

In conclusion I would not trust that site and instead use some other statistics software.


This looks like a bug. Also, the message at the top of the screen is a mis-statement of what a significant result means.

Instead of using a program like this, I recommend using standard statistical software such as R, SAS, SPSS etc. In R, you can use prop.test:

x <- c(18529, 18118)
n <- c(155590, 155592)

prop.test(x, n)

which gives a p value of 0.022, which is significant.

However, in most situations, this difference, while significant, will not be important. The two proportions are 0.119 and 0.116. Maybe there is some situation where this makes a difference, but I can't think of one, off-hand.

  • 1
    $\begingroup$ If a one sided test is what you want, that becomes prop.test(x=c(18529, 18118), n=c(155590, 155592), alternative = "less") or prop.test(x=c(18529, 18118), n=c(155590, 155592), alternative = "greater") $\endgroup$
    – Bernhard
    Nov 20 '19 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.