Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we want to optimize to reduce y output (a continuous reward with unknown min and max). Also, f(X) maps a different optimum y depending on some other continuous parameters Z, which are observable. Each day we can test a new X combination, and at the end of it we know the result.

I Could find out three posible solutions to this problem:

  1. The contextual multi-armed bandit problem solvable with Bayesian Optimization
  2. Variants of continuous reinforcement-learning/Q-learning
  3. Actor-critic methods

In the case when some of Z values are influenced by X (let's say only 2 non-related binary values), and other are external (3 real valued), which RL algorithm would fit better this problem, and how would I add these independent Z values to take them into account?

If all Z values are extenral, is multiarmed bandits the best approach?

  • 1
    $\begingroup$ None of the reinforcement learning or bandit-solving algorithms are really suitable for your black box problem. The bandit problem solver is maybe closest if $Z$ varies and is observable. You don't have a reinforcement learning problem unless choosing $X$ has a direct impact on the next value of $Z$. $\endgroup$ – Neil Slater Jan 9 '18 at 8:14
  • $\begingroup$ Let's say all the Z varies and are observable, all of them have an impact on the reward, but only some of them are influenced by X. In this case, which of these algorithms (or other) would be more suitable? I edited the question $\endgroup$ – freesoul Jan 9 '18 at 8:37
  • $\begingroup$ I would say the question is valid from the information acquisition perspective. $\endgroup$ – Karel Macek Jan 11 '18 at 0:54
  • $\begingroup$ @freesoul have you considered Gaussian process bandits? The variables Z can encode information such as smoothness of the function and the framework handles continuous x naturally. $\endgroup$ – combo Jan 14 '18 at 18:20

For small tasks you can use Bayesian Q-Learning https://ocs.aaai.org/Papers/AAAI/1998/AAAI98-108.pdf

For larger tasks you can use its variational version https://pdfs.semanticscholar.org/ce2e/0bd9135814f4018106bc31d87902b358e251.pdf

The core idea is to systematically try suboptimal x which shall pay off in the long term.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.