Consider two 2-arms bandits:
- average reward first arm is 2 euro, second arm 4 euro
- average reward first arm is 200 cents, second arm 400 cents
From my perspective, the bandits are exactly the same. However, UCB formula would give different results, because it uses absolute values instead of their proportions. Why? Have I missed some normalization step?
As a reminder, here is the formula (found, for example, here: https://www.cs.princeton.edu/courses/archive/fall16/cos402/lectures/402-lec22.pdf):
$A_t = argmax_a(Q_t(a) + sqrt(2 * ln(t) / N_a(t)))$