TLDR If the environment is rather complex (momentum, velocity, ...), but the action space is rather easy (left, right), is it not better to take multiple decisions into account before giving a reward instead of giving a reward for the last decision? Because previous bad decisions could result in state in which any decision would result in the game being lost, so the agent could learn the wrong behaviour...
I am talking about the cartpole environment from the open ai gym. https://gym.openai.com/envs/CartPole-v1/#id1
This environment seems to use some physics, such as the momentum of the cart. For example, if I move the cart to the right several times, then once to the left, it is still going to the right due to the momentum it has gained.
I am training a reinforcement learning algorithm (DQN) to use that model on. I currently use the sample network from https://github.com/keon/deep-q-learning/blob/master/dqn.py
The following confuses me:
Assuming the agent moves the pole right a lot. At some point the pole will probably get out of balance, and the episode will be over. Now the agent decideds to move the pole left at some point, after moving it right a lot. Due to the momentum, the episode might still be over, even though the agent now moved the pole left.
Because the agent only remembers the last state (it assumes the Markov propery holds), it will now remember that in this state, moving left seems to not be a good solution, since the reward was 0 because the game was lost. It does not remember that the game was actually lost because it moved right so much in the beginning. On the opposite, the first e.g. 10 times it moved right, it always got a reward because the game was not yet lost.
I'm not sure if the agent will somehow understand that the last action it did was not the 'wrong' action, rather the previous actions were the 'wrong' actions - it should not have moved the pole so much right..
Furthermore, will the agent eventually through trial and error learn that it should not have moved it right so much in the beginning? it probably will, but it might also get stuck and/or require a lot more training...
Would it not be a good idea to execute more steps before giving a reward? That way the agent would receive a negative reward for 'moving right a lot then left once' and perhaps more easily see the connection between multiple moves in a single direction and a bad reward?
- How should an agent with action size more than one learn about a good state?
- Reward attribution in deep q learning and texas holdem poker
- Is experience replay taking individual memory samples out of context? have partly answered my question. However if anyone still wants to answer my question, I would still appreciate it.