We have an events based tracking system for our website, with split testing built-in and we are using ABBA for the calculations.
The problem comes up when we are doing consecutive split tests. For example, we test X and Y, Y wins, so the next test is Y and Z. We are seeing, most of the time, that the conversion rate of Y differs in the two tests... Sometimes by as much as 50%.
We have a tool to track conversion rate over time, and I've included two graphs of conversion rate between two different tests of new and baseline. You can see that between the two tests, the previous new and the subsequent baseline differ by a significant amount even though they should theoretically be the same.
Both experiments have about 200,000 runs for new and baseline - 400k total - 800k over two experiments, and the P values on both are .0001. I'm just unable to explain why the conversion rate of "new" in test1 was 1.28% but the conversion rate of "baseline" in test2 was .85% - when they are exactly the same content.
I can certainly think of some things that would contribute to something like this: only catching part of a user cycle/time of day variations/seasonal variations/holidays, but my gut tells me that it shouldn't really make a 50% difference.
We have, however, eliminated the possibility of Googlebot and other bots affecting the results, as we don't even record events for bots.
Unfortunately, we don't have anything we can compare with. We do use Google analytics, but we only record our key metrics there, so there's no apples – apples.
The experiments were run immediately after each other, and I'm not aware of any special marketing campaigns (We don't do very much marketing, and no CPC). We are seeing some pretty good changes in traffic from certain countries, and I can definitely look into that to see if it could contribute... But, again, 50%?
Test 1: "New" wins
Test2: The old "New" is now "Baseline"...note the mismatch in conversion