I would like to know if it is possible to use the UCB bandit in a setting with continuous reward, particularly when the reward is zero-inflated. For instance, if I want to look at revenue or margin, most website visits are going to be zero. Those that are non-zero and thus have converted will have a relatively high reward (+20 and onward).
The confidence bound of UCB seems to be independent of the mean and therefore I expect the bounds to be erroneously small when applied to a continuous reward with mean > 1. Is this true? And is there an adjustment I can make such that UCB does work? If not, what other bandits are out there which are indeed suitable for continuous rewards?
Thanks in advance!