1
$\begingroup$

We have an events based tracking system for our website, with split testing built-in and we are using ABBA for the calculations.

The problem comes up when we are doing consecutive split tests. For example, we test X and Y, Y wins, so the next test is Y and Z. We are seeing, most of the time, that the conversion rate of Y differs in the two tests... Sometimes by as much as 50%.

We have a tool to track conversion rate over time, and I've included two graphs of conversion rate between two different tests of new and baseline. You can see that between the two tests, the previous new and the subsequent baseline differ by a significant amount even though they should theoretically be the same.

Both experiments have about 200,000 runs for new and baseline - 400k total - 800k over two experiments, and the P values on both are .0001. I'm just unable to explain why the conversion rate of "new" in test1 was 1.28% but the conversion rate of "baseline" in test2 was .85% - when they are exactly the same content.

I can certainly think of some things that would contribute to something like this: only catching part of a user cycle/time of day variations/seasonal variations/holidays, but my gut tells me that it shouldn't really make a 50% difference.

We have, however, eliminated the possibility of Googlebot and other bots affecting the results, as we don't even record events for bots.

Unfortunately, we don't have anything we can compare with. We do use Google analytics, but we only record our key metrics there, so there's no apples – apples.

The experiments were run immediately after each other, and I'm not aware of any special marketing campaigns (We don't do very much marketing, and no CPC). We are seeing some pretty good changes in traffic from certain countries, and I can definitely look into that to see if it could contribute... But, again, 50%?

Test 1: "New" wins Test 1 conversion

Test2: The old "New" is now "Baseline"...note the mismatch in conversion Test 2 conversion

Improve this question
$\endgroup$
3
  • $\begingroup$ 1. Does the graph show the aggregated / cumulative rate ? Or the rate per half-day-interval ? 2. 200.000 runs per test means 100000 per group, is this correct ? 3. I guess that the test has been stopped at the beginning of the sharp decline, is this correct ? $\endgroup$
    – steffen
    Jan 6, 2014 at 18:24
  • $\begingroup$ @steffen The graph shows the conversion rate per day, with the day almost completely isolated. The test is on a page with very little delayed conversion, so I'm pretty confident there isn't much contamination between days. Also, to clarify, new and baseline both get roughly 200,000 runs for a total of 400,000 for the experiment, and 800,000 for both experiments. The sharp decline is stopping the test. $\endgroup$
    – Jeremy T
    Jan 6, 2014 at 18:52
  • $\begingroup$ @tobias Thanks for the thoughts. I've updated with relevant details. $\endgroup$
    – Jeremy T
    Jan 6, 2014 at 18:54

1 Answer 1

Reset to default
3
$\begingroup$

I’ve witnessed a few such cases with some of my clients. When we investigated the discrepancies and re-ran experiments simultaneously we usually noticed that it came down to outside influences, usually the kind of traffic they were seeing on their pages. I’m assuming that you’re running an experiment on a website (as opposed to offline, in a store or so).

Do you have any other data source that you could compare conversion rates with? For instance, do you use an analytics tool like Google Analytics to track conversion rates? Did the two date ranges differ there too (if you compare the baseline)?

Did you run any special ad campaigns or change anything marketing-wise in the way you bring visitors to your pages between running those two experiments? I’ve seen sites that have started using content recommendation engines or started new search engine ad campaigns based on different keywords that brought more people to their site but ultimately decreased total conversion rate by even more than what you’re describing.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ Welcome to the site, @TobiasUrff. This isn't really an answer to the OP's question. It seems to be a set of questions for clarification. Please only use the "Your Answer" field to provide answers. I recognize it's frustrating, but you will be able to comment anywhere when your reputation >50. Since you are new here, you may want to read our tour page, which contains information for new users. $\endgroup$
    – gung - Reinstate Monica
    Jan 6, 2014 at 18:00
  • 1
    $\begingroup$ @gung To me it seems this post contains several potential answers. That some are phrased (politely) as interrogatives could be a little confusing, but they appear to be rather pointed and suggest possible answers. $\endgroup$
    – whuber ♦
    Jan 6, 2014 at 18:05
  • $\begingroup$ My apologies if I missed the point, Tobias. Perhaps you could make your potential answers a little more explicit? $\endgroup$
    – gung - Reinstate Monica
    Jan 6, 2014 at 18:15
  • $\begingroup$ Please remove the company name. I am pretty sure that the answer is valid without it and someone COULD see it as advertising ;) $\endgroup$
    – steffen
    Jan 6, 2014 at 18:31
  • $\begingroup$ Thanks for the feedback, @gung, whuber and steffen. I’ll take it into consideration in the future and will use comments whenever this seems more appropriate. I’m happy to see that, even though the form might have been wrong, jeremy-tunnell has since updated his question with some more details on the pointers I gave. steffen, I have removed my employer‘s company name. It wasn’t intended as advertising but rather to imply that I had visibility into a good amount of different organizations, websites and experiments where this happened, but I understand your concerns. :) $\endgroup$
    – Tobias Urff
    Jan 6, 2014 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.