# Reinforce Learning for environment which cannot be affected by agent

The model of RL is defined as P^a_ss', the action space is continuous. In order to make the agent knows that the env would behave it own ways regardless what the agent does, what would I do?

It is also desirable to learn the state transition of the env, would RL at all be suffice for the job? If yes, the env has only one continuous variable x_0 in in observation space, and a numberous of hidden factors x_1, x_2, ... that affect x_0; should x_1, x_2, ... be in the observation space too? If no, what would I do next beside Recurrent Neural Network?

EDIT:

How hidden is my hidden factors? Could they realistically be part of a HMM?

Prior belief that they affect x_0, but not sure if x_1...x_n is an exhaustive list. All of them are visible, but how they affect x_0 itself can very well be a time series fitting problem. In order to model them as HMM, we have to know how long the timestep is. I guess, and it is really a guess, that HMM require fixed timestep length, which might or might not be the case here.

The env would behave its own ways regardless what the agent does, but the action of the agent will affect its reward.

The goal here is learning optimal policy as well as the state transition.

• How hidden are your "hidden factors". Can they actually be measured/observed by an agent? Could they realistically be part of a HMM (similar idea to RNN, less generic, but an explicit state model, so easier to manage in RL context)? When you say " the env would behave it own ways regardless what the agent does", do you mean only that the action does not affect state transition $s \rightarrow s'$, but still influences immediate reward $r$? Or is reward also independent of action taken, and you simply want to learn the state transition in isolation of optimising reward? – Neil Slater Jul 5 '18 at 10:24
• @NeilSlater please see edit. Awesome questions! – Minh Triet Jul 5 '18 at 20:31

In order to make the agent knows that the env would behave it own ways regardless what the agent does, what would I do?

You can technically use RL to model this problem, but since both immediate reward r and next rewards r' do not depend on agents' actions, there is no chance that state-action transitions will learn to maximize long-term reward R.

should x_1, x_2, ... be in the observation space too?

If adding these factors contribute to better convergence (learning to maximize R), then they should be added. It can be that all their information has been summarized to the continuous variable, and then they are abundant.

If no, what would I do next beside RNN?

What is that?

• RNN is Recurrent Neural Network. Sorry for the confusion – Minh Triet Jul 5 '18 at 9:05
• I got it. Using any model in your case is questionable, since the environment response to agent actions should not be 100% stochastic. – Alexey Burnakov Jul 5 '18 at 9:08

So you have a situation where:

• Actions do have influence on the immediate, one-step rewards
• Actions do not have influence on state transitions, so also do not have influence on future returns other than the immediate one-step rewards.

This means that you do not have a "standard" Reinforcement Learning problem, you have what's called a Multi-Armed Bandit problem. More specifically, since you do appear to still have some concept of "states" (which I assume may or may not contain somewhat useful information about which actions might be optimal for one-step rewards given the current states), you have a Contextual Multi-Armed Bandit problem (where "states" are more commonly referred to as "context (vectors").

These really are just special cases of the more general Reinforcement Learning setting. Any algorithm that can handle the more typical RL problem can in theory also handle (contextual) MAB problems. They're probably not the best algorithms though. If you try to throw a standard RL algorithm at a MAB problem, the RL algorithm will "think" that it may have influence on state transitions, and therefore "try to learn" (very informal language here) about that too (even if it's impossible/useless).

There also are dedicated algorithms for (contextual) MAB problems. These do not waste any learning effort trying to learn about state transitions which they have no influence over anyway, they just try to learn about the reward distributions per arm (= action) given the current context (= state).

As for which variables you should include in your observation space, I'd recommend including as many observable variables as possible if there's a chance that they're somewhat informative, at least if that doesn't lead to computational problems for you. Forgetting to include an informative feature tends to be much more harmful than accidentally adding an uninformative (for example, random) feature. This is just what tends to be my experience, it's impossible to tell for sure for any specific situation. If you want to be sure, try different alternatives and evaluate performance for your specific case.

• Thank you @DennisSoermers. As for not learning state transition that the agent cannot affect, I am unsure. The ideal case is: The agent knows when to play and when to wait by predicting the next state and reward, according to the visible context vector. – Minh Triet Jul 6 '18 at 21:08
• @MinhTriet That can be modelled in the same way by treating "wait" as a "null"-action that always gives a reward of $0$ – Dennis Soemers Jul 7 '18 at 8:33