AB Multi-variate testing - when can I eliminate worst performing variants? I'm running AB tests on my website homepage, with 8 different variations. You can read about the purpose of the test here if you need to (not essential) - http://westiseast.co.uk/blog/ab-split-testing-a-promise-ogilvy/
My question is this - at what point can I remove the worst performing variants from the test? 
Can I remove the worst performing variant when it achieves statistical significance when compared to the BEST performing variant? Or when it achieves statistical significance compared to the 2nd worst performing variant?
I'd like to continually refine the test, and removing the worst performing examples improves the traffic going to the 'better' ones.
Many thanks for your help!
 A: It is hard to answer this a priori, but you should take a look at "multi-armed bandit" methods.  In essence, you have N items, and do not know the distribution of the returns.  Unlike standard sequential analysis, where the goal is to test two (or more) random values, in this case you're also looking at the returns associated with a particular variant & interested in maximizing long-term results.
A major example of how this is done is Google's AdWords, which may take a number of different text variations for ads for a given site, using the same keywords, targeting, etc.  The variants will be shown, over time, to many different users and those that maximize expected revenue (to Google :)) will be given a higher probability of being shown than those that have lower expected returns.
In your case, not enough information is given to give you a precise answer, but the bandit methods will be a good starting point.  Moreover, without actually applying a particular algorithm, it is hard to say how these will pan out.  I'd probably try several algorithms just to see how well these do in your context, for your data, users, and objective functions.
