I have a problem where, effectively, my slot machines have very low payout probability (on the order of 1% for the "best" slot machines) and my goal is to minimize the number of actions to get one reward. I'm wondering if my goal is still equivalent to maximizing long term reward (as in the original formulation). At a higher level, I'm wondering which of the available tools for solving Bernoulli bandit are best to use in my scenario, or even if an altogether different approach would be more appropriate.

I should mention, I have an initial guesstimate for the reward probability of each slot machine (and they are distinct) and I can incorporate that into whatever method I use.

  • $\begingroup$ You may as well say all the winning rewards are of equal value so wanting in the long term to minimise expected gaps is the same as wanting to maximise the expected number of rewards which becomes the same as wanting to maximise the expected amounts of rewards. $\endgroup$
    – Henry
    Oct 6, 2023 at 15:00
  • $\begingroup$ @Henry I was thinking about it this way too. I just felt a little unsteady about it for some reason. I think you saying it "out loud" helps. $\endgroup$ Oct 6, 2023 at 15:03
  • $\begingroup$ But now I'm just wondering which is the best approach to use with sparse rewards. $\endgroup$ Oct 6, 2023 at 15:03

1 Answer 1


my goal is to minimize the number of actions to get one reward

That seems to change to setting quite a bit. Does that mean that you are playing until you get one reward, and then the episode ends? (And your goal is to minimize the amount of rounds before the first reward is generated.) If that's the case, then the awerage reward observed from all arms is 0 for the whole game before end. Thus, the optimal strategy would be to play the arm with smallest amount of attempts in each round. (i.e. cycle between all arms, until the first reward is generated.)

  • $\begingroup$ I should mention that I can at least start with a prior quesstimate for the payout probability of each arm. I think I said it but I didn't say I'd incorporate it as an intial prior. I was thinking to use a Bernoulli distribution for them. $\endgroup$ Oct 6, 2023 at 14:56
  • $\begingroup$ Suppose you were planning to play a total of $n$ times and want to maximise the number of rewards you received. Would you initially cycle until the first reward and then stick with that arm forever? $\endgroup$
    – Henry
    Oct 6, 2023 at 14:56
  • $\begingroup$ @Henry If you restate the objective as maximizing reward then I feel like the right thing would be to revert to using the well characterized methods already out there like Thompson sampling or UCB. $\endgroup$ Oct 6, 2023 at 14:58
  • $\begingroup$ I think that is the sensible approach, but the interesting question here is the sparse occurrence of rewards (even if, when they happen, they are of the same value) $\endgroup$
    – Henry
    Oct 6, 2023 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.