I would like to know if it is possible to use the UCB bandit in a setting with continuous reward, particularly when the reward is zero-inflated. For instance, if I want to look at revenue or margin, most website visits are going to be zero. Those that are non-zero and thus have converted will have a relatively high reward (+20 and onward).

The confidence bound of UCB seems to be independent of the mean and therefore I expect the bounds to be erroneously small when applied to a continuous reward with mean > 1. Is this true? And is there an adjustment I can make such that UCB does work? If not, what other bandits are out there which are indeed suitable for continuous rewards?

Thanks in advance!

  • 1
    $\begingroup$ Hi, the standart / most common (afaik) form of UCB assume that the reward are generated from normal distribution (resp. any distribution for which the so-called sub-gaussian assumption holds). So it assume that rewards are continuous. Do you have any specific version of UCB in mind? The problem would be that UCB need to make some assumptions about the distribution, which will be probably violated in your case. $\endgroup$ – Jakub Koubele Dec 9 '19 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.