0
$\begingroup$

We want to pose one problem as a multi-armed bandit setting. The issue is that some of the arms are very risky with potentially undesirable effects (or not). Is there a way to do a risk-aware exploration where you stop OR reduce exploration of particular arm(s) if the rewards observed are of concern while still exploring & exploiting other 'safe' arms?

I can think of the softmax multi-armed bandits, but maybe there is something smarter than that.

$\endgroup$
1
$\begingroup$

Yes, you can! One way of taking into account the risk is to use distributional reinforcement learning, where you learn the entire distribution of the rewards (in each state) rather than just keeping track of the expected rewards as is standard in reinforcement learning. Once you have access to the distribution of rewards for each arm, you can design whatever policy you desire that takes into account the risk factor.

Consider this simple example. Say, you want the agent to avoid all options that can lead to negative rewards. If you have access to the distributions of values for each arm, you can map the value distributions to a (scalar) metric that is not the expected reward, but rather: $V(x_i) \rightarrow -\infty$ if $\int_{-\infty}^{0} p(V(x_i)) dV > \epsilon$, else $V(x_i) = \int V(x_i) p(V(x_i)) dV = mean(V(x_i))$, where $\epsilon$ is an arbitrarily small number. In words, for all options that have negative rewards, you artificially set the scalar value which you will use to choose an action to $-\infty$ while using the mean value for all other options that do not have negative rewards.

There are two proposed methods of learning the value distribution rather than the expected values that I know of. One of them is from Dabney et al. 2020, and the other one from Tano et al. 2020. The code for both is linked in the respective papers, but for a quick breakdown: Deepmind's code uses a mathematically simple code to keep track of each expectile of the value distribution individually, but it is non-local, which can be viewed as slightly problematic for the implementing the updates (especially if using neural nets for the value function), as opposed to Tano's Laplace code, which is local.

$\endgroup$
3
  • $\begingroup$ Interesting. How difficult is it to learn the distribution as opposed to a simple mean? I am using plain multi-armed bandits (not even contextual) that are just averaging the rewards. Once you learn the distribution, how do you know that it is realistic and/or keep exploring the 'dangerous' arms (just in case). Doesn't this approach introduce a strong bias to the exploration strategy (cause the 'risky' arms will never be pulled)? $\endgroup$
    – d56
    Sep 3 at 14:02
  • $\begingroup$ It takes more updates to converge to the full distribution as opposed to just the mean, if that's what you mean :). Both the papers I have linked have convergence guarantees in the limit of infinite samples. Once you have the true distribution, it all depends on your policy. The very simple policy that I described in my answer will never pull risky arms, but you can design another one that would choose the risky arms once in a while. $\endgroup$
    – lonelyOrca
    Sep 9 at 13:46
  • $\begingroup$ And if you're talking about the difficulty of implementing a distributional RL algorithm, you can already find the code for it integrated into the rainbow deep RL github repo: github.com/Kaixhin/Rainbow $\endgroup$
    – lonelyOrca
    Sep 9 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.