Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we want to optimize to reduce y output (a continuous reward with unknown min and max). Also, f(X) maps a different optimum y depending on some other continuous parameters Z, which are observable. Each day we can test a new X combination, and at the end of it we know the result.

I could find out three possible solutions to this problem:

  1. The contextual multi-armed bandit problem solvable with Bayesian Optimization
  2. Variants of continuous reinforcement-learning/Q-learning
  3. Actor-critic methods

In the case where some Z values are influenced by X (let's say only 2 non-related binary values), and other are external (3 real valued), which RL algorithm would fit better this problem, and how would I add these independent Z values to take them into account?

If all Z values are external, is multiarmed bandits the best approach?

  • 1
    $\begingroup$ None of the reinforcement learning or bandit-solving algorithms are really suitable for your black box problem. The bandit problem solver is maybe closest if $Z$ varies and is observable. You don't have a reinforcement learning problem unless choosing $X$ has a direct impact on the next value of $Z$. $\endgroup$ Commented Jan 9, 2018 at 8:14
  • $\begingroup$ Let's say all the Z varies and are observable, all of them have an impact on the reward, but only some of them are influenced by X. In this case, which of these algorithms (or other) would be more suitable? I edited the question $\endgroup$
    – freesoul
    Commented Jan 9, 2018 at 8:37
  • $\begingroup$ I would say the question is valid from the information acquisition perspective. $\endgroup$ Commented Jan 11, 2018 at 0:54
  • $\begingroup$ @freesoul have you considered Gaussian process bandits? The variables Z can encode information such as smoothness of the function and the framework handles continuous x naturally. $\endgroup$
    – combo
    Commented Jan 14, 2018 at 18:20

1 Answer 1


For small tasks you can use Bayesian Q-Learning.

For larger tasks, you can use its variational version.

The core idea is to systematically try suboptimal x which shall pay off in the long term.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.