# What are the differences between contextual bandits, actor-citric methods, and continuous reinforcement learning?

Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we want to optimize to reduce y output (a continuous reward with unknown min and max). Also, f(X) maps a different optimum y depending on some other continuous parameters Z, which are observable. Each day we can test a new X combination, and at the end of it we know the result.

I Could find out three posible solutions to this problem:

In the case when some of Z values are influenced by X (let's say only 2 non-related binary values), and other are external (3 real valued), which RL algorithm would fit better this problem, and how would I add these independent Z values to take them into account?

If all Z values are extenral, is multiarmed bandits the best approach?

• None of the reinforcement learning or bandit-solving algorithms are really suitable for your black box problem. The bandit problem solver is maybe closest if $Z$ varies and is observable. You don't have a reinforcement learning problem unless choosing $X$ has a direct impact on the next value of $Z$. – Neil Slater Jan 9 '18 at 8:14
• Let's say all the Z varies and are observable, all of them have an impact on the reward, but only some of them are influenced by X. In this case, which of these algorithms (or other) would be more suitable? I edited the question – freesoul Jan 9 '18 at 8:37
• I would say the question is valid from the information acquisition perspective. – Karel Macek Jan 11 '18 at 0:54
• @freesoul have you considered Gaussian process bandits? The variables Z can encode information such as smoothness of the function and the framework handles continuous x naturally. – combo Jan 14 '18 at 18:20

The core idea is to systematically try suboptimal x which shall pay off in the long term.