I am experimenting with the multi-armed bandit algorithms (namely: epsilon greedy, decaying epsilon greedy, optimistic initial value, upper confidence interval, and Thompson sampling).
My reward is continuous and so I use a Gaussian distribution for the Thompson sampling (I'm not sure if this specifically can be a problem, but my question below is rather general).
All the examples and tutorials I could find (including this related question here) deal with either binary output (win/lose) or rewards with standard normal distribution.
However, playing with the numbers I noticed that nearly all the algorithms either fail, or become very sensitive to the hyper-parameters (if relevant) when the rewards are fairly high (>50), very low (~0.001) or has large variance.
I read several comments on related issues suggesting scaling the data, but this is sometimes not feasible in practice.
My question then is: is this the limit of these methods, or is there something fundamental that I am missing here?
Thank you very much!
Note: overall, the epsilon algorithms are the most 'resilient' in my experience here.