Bandit learning with biased and unbiased data

Question

I have an online experimentation setup with incoming customers split into 3 groups:

• Random (all arms are applied equally) 20%
• Model-based (an existing, optimal strategy is run) 40%
• MAB (Multi-armed bandit): this is a new setup. 40%

The MAB traffic split is introduced to compare with the existing model-based approach. I would like to ask:

1. Since Random traffic is always there, it can be used as exploration and MAB just needs to run exploitation on its own traffic. Does it mean off-policy learning? Any keyword for further search is appreciated.
2. Which data traffic should be used to update MAB? I feel Random and MAB make sense but not sure about Model-based traffic, as it is biased with another policy.

Thanks for reading my question.

Answer 1

I just read your question. Maintaining a random variant is always interesting because it is a great feedback input without any presentation bias. I am working in a company as a ML Engineer in which I have developed a MAB based on Thompson Sampling and I came across a very similar situation to the one you are talking about. It seems logical to think that the random variant is the right one to feed our MAB because of its lack of bias and exploration. But, 90% of the MABs I have seen have already some exploration! Many of them are based on the explore-exploit tradeoff. So, to answer your second question, if your MAB has an exploration part, you can use its feedback as input (you probably need to do some transformation to the data or weighting in some way).

Regarding your first question, if you force the MAB to exploit, it is no longer a MAB. As I said before, these algorithms are based on explore-exploit. If you remove one of those parts, you are doing the same thing you would do if you had a model that maximised the probability of having a favourable event after the MAB output. It would be the same thing.

I hope you find it helpful or that you have found the solution to your questions!