What could cause different conversion rates for the same content in separate split tests? We have an events based tracking system for our website, with split testing built-in and we are using ABBA for the calculations.
The problem comes up when we are doing consecutive split tests. For example, we test X and Y, Y wins, so the next test is Y and Z. We are seeing, most of the time, that the conversion rate of Y differs in the two tests... Sometimes by as much as 50%.
We have a tool to track conversion rate over time, and I've included two graphs of conversion rate between two different tests of new and baseline. You can see that between the two tests, the previous new and the subsequent baseline differ by a significant amount even though they should theoretically be the same.
Both experiments have about 200,000 runs for new and baseline - 400k total - 800k over two experiments, and the P values on both are .0001.  I'm just unable to explain why the conversion rate of "new" in test1 was 1.28% but the conversion rate of "baseline" in test2 was .85% - when they are exactly the same content.
I can certainly think of some things that would contribute to something like this: only catching part of a user cycle/time of day variations/seasonal variations/holidays, but my gut tells me that it shouldn't really make a 50% difference.
We have, however, eliminated the possibility of Googlebot and other bots affecting the results, as we don't even record events for bots.
Unfortunately, we don't have anything we can compare with. We do use Google analytics, but we only record our key metrics there, so there's no apples – apples.
The experiments were run immediately after each other, and I'm not aware of any special marketing campaigns (We don't do very much marketing, and no CPC). We are seeing some pretty good changes in traffic from certain countries, and I can definitely look into that to see if it could contribute... But, again, 50%?
Test 1: "New" wins

Test2: The old "New" is now "Baseline"...note the mismatch in conversion

 A: I’ve witnessed a few such cases with some of my clients. When we investigated the discrepancies and re-ran experiments simultaneously we usually noticed that it came down to outside influences, usually the kind of traffic they were seeing on their pages. I’m assuming that you’re running an experiment on a website (as opposed to offline, in a store or so).
Do you have any other data source that you could compare conversion rates with? For instance, do you use an analytics tool like Google Analytics to track conversion rates? Did the two date ranges differ there too (if you compare the baseline)?
Did you run any special ad campaigns or change anything marketing-wise in the way you bring visitors to your pages between running those two experiments? I’ve seen sites that have started using content recommendation engines or started new search engine ad campaigns based on different keywords that brought more people to their site but ultimately decreased total conversion rate by even more than what you’re describing.
