# Multi-armed bandit vs AB testing

I'm trying to understand the difference between AB Testing and Multi armed bandit and when we should use one of the other. It seems to me that Multi-armed bandit algorithms are statistically valid and require fewer data points. Can someone help me understand when you should use one over the other?

# The Difference

I'm going to assume you are interested in this topic from a website design perspective.

In an A/B test, you choose layout A half the time and layout B the other half. You record how much revenue is collected under each layout for $$n$$ visitors. Then you do a statistical test to determine if layout A or layout B had a statistically significantly higher revenue, and then deploy that layout. In bandit language, you had pure exploration period (trying out each option and seeing the revenue) for $$n$$ time steps and then a pure exploitation period afterwards (deploying the layout estimated to be the best).

In a multi-armed bandit setting, the decision of when to show layout A or layout B is chosen by the algorithm. Instead of a pure exploration period followed by a pure exploitation period, you can continually balance exploration and exploitation. The algorithm will continually adjust its display percentages based on:

1. An estimate for how much revenue the layout will get, based on the data collected so far. The higher the estimate the more the algorithm wants to display that layout.
2. How confident we are in that estimate, based on how much data we've collected so far and its variability. The less confidence in the estimate, the more the algorithm wants to display that layout in order to collect more data about it.

So the main difference comes down to:

• A/B tests tries out each option a constant % of the time, then at some point it's decided which one is best and that option is deployed
• Bandit algorithms can continually adjust how often to display each option based on how they're performing. How exactly this works is dependent on the bandit algorithm that you choose.

Something to take note of is that multi-armed bandit is the name of the problem of sequential decision making under uncertainty and there are various algorithms that provide provably good solutions to this problem. A/B testing is actually the same thing as the explore-then-commit bandit algorithm. It doesn't necessarily make sense to look at "A/B Test vs. Multi-armed Bandits" since A/B tests are also a multi-armed bandit algorithm.

# Which one to choose?

I don't think there's a good answer to this. Since A/B tests are a bandit algorithm, your question reduces to "Which multi-armed bandit algorithm should I choose?" and this is dependent on the problem you are trying to solve. What do your rewards look like? Is your problem stationary? Is context important? Will you be adding new options all the time? The bandit setting is general enough that you can probably find a bandit algorithm to fit your specific problem. This includes non-stationarity, contextual bandits, ranking bandits, cascading bandits, etc.

TL;DR: A/B Testing is also a bandit algorithm. There's no good rule of thumb (that I'm aware of) for choosing which algorithm to use in this setting. I suggest formalizing your problem and deciding what's important from a time and business perspective, then seeing which algorithms match up with your needs. If it's a simple scenario it's probably most reasonable to go with a simple A/B test or $$\epsilon$$-greedy bandit algorithm. For something intricate, high-volume or high-value, I'd go with a fancier bandit algorithm that's tailored to your situation.

Assuming we are talking about traffic to a website, AB testing refers the method whereby traffic is split 50/50 between two different pages (or options, images, or whatever you are studying). So exactly half of your users would see, say, Page A and the other half would see Page B. You then use the results (e.g. purchases) to run your statistical tests. In multi-armed bandit, for a small percentage of the time (usually 10%), you'd split your traffic evenly between Page A and Page B just as you would in AB Testing. For the other 90% of the traffic, you'd divert users to the best performing page (if pages are tied, you'd randomly select a page to direct the user to).

AB testing has a potentially high performance loss because you are not directing users to the best performing Page most of the time. On the other hand, it takes very little time to gather enough data points (visitors) to both pages to be able to perform statistical tests for differences in performance more quickly compared to the multi-bandit approach. Under the multi-bandit approach, only 10% of the time are you randomly sending traffic randomly to Page A or Page B, and within that 10%, roughly only 50% (or 5% of your total traffic) is going to the initially under-performing page.

As a result, multi-bandit methods are best suited in those cases where you can wait longer periods before knowing which page is better performing. Multi-bandit approach is usually reserved for continuous optimization of existing pages, whereas AB is best suited for brand new pages that require experimentation.

You will find a nice simulation study that should help you understand the differences between the two methods here.

• Multiarmed is a type of problem. en.m.wikipedia.org/wiki/Multi-armed_bandit. What you are describing is a possible solution, epsilon greedy. An alternative would be Thompson sampling which would start 50/50 and then update according to how confident you are that one arm is actually better... Commented Dec 19, 2018 at 19:44