We're using Z-test of proportions to calculate the significance of the difference in conversion rates between two variants of the same website.
While it works as expected most of the time, we recently ran an A/A test, meaning we deliberately used the same variant of the website, in order to check whether it yields significant results. The test yielded a significant result.
We realize that this can happen from time to time, so we ran the test again, and we again reached our significance threshold p<0.05. We finally tried a simulated scenario where we the data we supplied wasn't user data, but randomly generated data, and it ended up significant again.
An important thing to note is that the difference in proportions is very low. We have variant A winning with 10.8% conversion vs 10.7% conversion, with a sample size of around 300000.
We tried to use Chi-Square instead, but it doesn't offer much improvement.
What we're trying to figure out is whether our approach is correct and if not what alternatives do we have?