Reinforce Learning for environment which cannot be affected by agent The model of RL is defined as P^a_ss', the action space is continuous. In order to make the agent knows that the env would behave it own ways regardless what the agent does, what would I do?
It is also desirable to learn the state transition of the env, would RL at all be suffice for the job? If yes, the env has only one continuous variable x_0 in in observation space, and a numberous of hidden factors x_1, x_2, ... that affect x_0; should x_1, x_2, ... be in the observation space too? If no, what would I do next beside Recurrent Neural Network?
EDIT: 
How hidden is my hidden factors? Could they realistically be part of a HMM?
Prior belief that they affect x_0, but not sure if x_1...x_n is an exhaustive list. All of them are visible, but how they affect x_0 itself can very well be a time series fitting problem. In order to model them as HMM, we have to know how long the timestep is. I guess, and it is really a guess, that HMM require fixed timestep length, which might or might not be the case here. 
The env would behave its own ways regardless what the agent does, but the action of the agent will affect its reward.
The goal here is learning optimal policy as well as the state transition.
 A: So you have a situation where:


*

*Actions do have influence on the immediate, one-step rewards

*Actions do not have influence on state transitions, so also do not have influence on future returns other than the immediate one-step rewards.


This means that you do not have a "standard" Reinforcement Learning problem, you have what's called a Multi-Armed Bandit problem. More specifically, since you do appear to still have some concept of "states" (which I assume may or may not contain somewhat useful information about which actions might be optimal for one-step rewards given the current states), you have a Contextual Multi-Armed Bandit problem (where "states" are more commonly referred to as "context (vectors").
These really are just special cases of the more general Reinforcement Learning setting. Any algorithm that can handle the more typical RL problem can in theory also handle (contextual) MAB problems. They're probably not the best algorithms though. If you try to throw a standard RL algorithm at a MAB problem, the RL algorithm will "think" that it may have influence on state transitions, and therefore "try to learn" (very informal language here) about that too (even if it's impossible/useless).
There also are dedicated algorithms for (contextual) MAB problems. These do not waste any learning effort trying to learn about state transitions which they have no influence over anyway, they just try to learn about the reward distributions per arm (= action) given the current context (= state).
As for which variables you should include in your observation space, I'd recommend including as many observable variables as possible if there's a chance that they're somewhat informative, at least if that doesn't lead to computational problems for you. Forgetting to include an informative feature tends to be much more harmful than accidentally adding an uninformative (for example, random) feature. This is just what tends to be my experience, it's impossible to tell for sure for any specific situation. If you want to be sure, try different alternatives and evaluate performance for your specific case.
A: 
In order to make the agent knows that the env would behave it own ways
  regardless what the agent does, what would I do?

You can technically use RL to model this problem, but since both immediate reward r and next rewards r' do not depend on agents' actions, there is no chance that state-action transitions will learn to maximize long-term reward R.

should x_1, x_2, ... be in the observation space too?

If adding these factors contribute to better convergence (learning to maximize R), then they should be added. It can be that all their information has been summarized to the continuous variable, and then they are abundant.

If no, what would I do next beside RNN?

What is that?
