I am running an A/B test for our site. And get below stats:
Group A: Visitors: 155,590; Conversion: 18,529
Group B: Visitors: 155,592; Conversion: 18,118
And I am using the tool on this website to determine if the test result is significant or not. https://abtestguide.com/calc/
The P-value is 0.9889 which is way higher than 0.05, but the tool still says it's significant. I am very confused. I thought only if the p-value is smaller than 0.05, I can say it's significant (assuming 95% confidence).
I have entered your numbers into the form and the result is shown to be significant for the two-sided test but not for the one-sided test whilst the p-value is constant. If you change A and B values both tests (one- and two-tailed) are significant and the p-value becomes < 0.05.
The p-value shown seems to be that of a one-sided test with the order in which you give the data determines the direction of that test, even if you perform a two-sided tests. Two-sided tests are usually considered standard and I consider it bad practice, to show one-sided p-values when a two-sided test is ticked.
In conclusion I would not trust that site and instead use some other statistics software.
This looks like a bug. Also, the message at the top of the screen is a mis-statement of what a significant result means.
Instead of using a program like this, I recommend using standard statistical software such as R, SAS, SPSS etc. In R, you can use prop.test:
x <- c(18529, 18118)
n <- c(155590, 155592)
prop.test(x, n)
which gives a p value of 0.022, which is significant.
However, in most situations, this difference, while significant, will not be important. The two proportions are 0.119 and 0.116. Maybe there is some situation where this makes a difference, but I can't think of one, off-hand.
prop.test(x=c(18529, 18118), n=c(155590, 155592), alternative = "less") or prop.test(x=c(18529, 18118), n=c(155590, 155592), alternative = "greater")
$\endgroup$