4
$\begingroup$

I'm running AB tests on my website homepage, with 8 different variations. You can read about the purpose of the test here if you need to (not essential) - http://westiseast.co.uk/blog/ab-split-testing-a-promise-ogilvy/

My question is this - at what point can I remove the worst performing variants from the test?

Can I remove the worst performing variant when it achieves statistical significance when compared to the BEST performing variant? Or when it achieves statistical significance compared to the 2nd worst performing variant?

I'd like to continually refine the test, and removing the worst performing examples improves the traffic going to the 'better' ones.

Many thanks for your help!

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ Aha! I see that @whuber has already added sequential-analysis as a tag. :) That's the right path. $\endgroup$
    – Iterator
    Aug 15, 2011 at 13:56
  • $\begingroup$ CV needs "multi-armed-bandit" as a tag. Unfortunately, I can't add it. $\endgroup$
    – Iterator
    Aug 16, 2011 at 3:16
  • $\begingroup$ @Iterator the tag reinforcement-learning tag is available and should be appropriate, too. Regarding sequential analysis: Yes, I definitely think that is is the right way to perform AB-Tests in a statistical correct way, but by looking around I do not have the impression that anyone of the mainstream AB-test-users knows this (or cares). Dropping sequential analysis, reinforcement-learning is certainly the more practical way to go. $\endgroup$
    – mlwida
    Aug 16, 2011 at 6:23
  • $\begingroup$ @steffen: You raise an interesting point. I didn't recommend RL because the author was more interested in sampling and testing than model improvement. Although it may seem that the "lay" literature on A/B testing ignores sequential analysis, there is more than enough practical work accessible on it, though it takes a bit more effort to grapple with. Bandit methods are approximations that the OP may find useful. The issue with RL is that that is a very large canopy term that can lead the unwary into a large theoretical forest, far removed from anything useful (aka NIPS). $\endgroup$
    – Iterator
    Aug 16, 2011 at 11:29
  • $\begingroup$ (continued) If the question were posted by someone exhibiting a sufficiently high level of stats & ML familiarity, I would lean more toward suggesting RL, especially if the question focused on improving a model while sampling judiciously. As the question appears more related to sampling judiciously, I lean toward simple bandit algos that the author could begin to apply immediately. $\endgroup$
    – Iterator
    Aug 16, 2011 at 11:31

1 Answer 1

Reset to default
3
$\begingroup$

It is hard to answer this a priori, but you should take a look at "multi-armed bandit" methods. In essence, you have N items, and do not know the distribution of the returns. Unlike standard sequential analysis, where the goal is to test two (or more) random values, in this case you're also looking at the returns associated with a particular variant & interested in maximizing long-term results.

A major example of how this is done is Google's AdWords, which may take a number of different text variations for ads for a given site, using the same keywords, targeting, etc. The variants will be shown, over time, to many different users and those that maximize expected revenue (to Google :)) will be given a higher probability of being shown than those that have lower expected returns.

In your case, not enough information is given to give you a precise answer, but the bandit methods will be a good starting point. Moreover, without actually applying a particular algorithm, it is hard to say how these will pan out. I'd probably try several algorithms just to see how well these do in your context, for your data, users, and objective functions.

Improve this answer
$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy