We're using Z-test of proportions to calculate the significance of the difference in conversion rates between two variants of the same website.

While it works as expected most of the time, we recently ran an A/A test, meaning we deliberately used the same variant of the website, in order to check whether it yields significant results. The test yielded a significant result.

We realize that this can happen from time to time, so we ran the test again, and we again reached our significance threshold p<0.05. We finally tried a simulated scenario where we the data we supplied wasn't user data, but randomly generated data, and it ended up significant again.

An important thing to note is that the difference in proportions is very low. We have variant A winning with 10.8% conversion vs 10.7% conversion, with a sample size of around 300000.

We tried to use Chi-Square instead, but it doesn't offer much improvement.

What we're trying to figure out is whether our approach is correct and if not what alternatives do we have?

  • 1
    $\begingroup$ With that sample size almost any difference however miniscule is going to achieve an arbitrary level of statistical significance. $\endgroup$
    – mdewey
    Dec 21 '20 at 17:22
  • $\begingroup$ Agree with @mdewey. Also, you need to decide how large a difference in conversion rate is of practical importance. For example, is the difference between 10.8% and 10.7% meaningful. $\endgroup$
    – BruceET
    Dec 21 '20 at 21:44
  • $\begingroup$ Thanks, @BruceET that makes sense. Suppose we arbitrarily decide that a difference that small is insignificant, would the proper way to go about this would be to check if the difference first meets the threshold and then check significance? Or is there any way we can modify the test to be more strict? $\endgroup$ Dec 22 '20 at 9:51
  • $\begingroup$ In a formal report. I'd say the sample is so large that the difference is technically 'statistically significant' (at whatever level), but for practical purposes too small to be of importance. $\endgroup$
    – BruceET
    Dec 22 '20 at 21:03
  • $\begingroup$ Two questions: 1) Can you describe how the "randomly generated data" is generated? 2) Many things can lead to an A/A test producing a statistically significant result, apart from what @mdewey mentioned, a common source is that one A is not actually equal to the other A (e.g. one requires a redirect, the other doesn't) - is that accounted for? $\endgroup$
    – B.Liu
    Dec 22 '20 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.