Consider more than just a single state-action pair before giving a reward in deep q learning?

Question

TLDR If the environment is rather complex (momentum, velocity, ...), but the action space is rather easy (left, right), is it not better to take multiple decisions into account before giving a reward instead of giving a reward for the last decision? Because previous bad decisions could result in state in which any decision would result in the game being lost, so the agent could learn the wrong behaviour...

---

I am talking about the cartpole environment from the open ai gym. https://gym.openai.com/envs/CartPole-v1/#id1

This environment seems to use some physics, such as the momentum of the cart. For example, if I move the cart to the right several times, then once to the left, it is still going to the right due to the momentum it has gained.

I am training a reinforcement learning algorithm (DQN) to use that model on. I currently use the sample network from https://github.com/keon/deep-q-learning/blob/master/dqn.py

The following confuses me:

Assuming the agent moves the pole right a lot. At some point the pole will probably get out of balance, and the episode will be over. Now the agent decideds to move the pole left at some point, after moving it right a lot. Due to the momentum, the episode might still be over, even though the agent now moved the pole left.

Because the agent only remembers the last state (it assumes the Markov propery holds), it will now remember that in this state, moving left seems to not be a good solution, since the reward was 0 because the game was lost. It does not remember that the game was actually lost because it moved right so much in the beginning. On the opposite, the first e.g. 10 times it moved right, it always got a reward because the game was not yet lost.

I'm not sure if the agent will somehow understand that the last action it did was not the 'wrong' action, rather the previous actions were the 'wrong' actions - it should not have moved the pole so much right..

Furthermore, will the agent eventually through trial and error learn that it should not have moved it right so much in the beginning? it probably will, but it might also get stuck and/or require a lot more training...

Would it not be a good idea to execute more steps before giving a reward? That way the agent would receive a negative reward for 'moving right a lot then left once' and perhaps more easily see the connection between multiple moves in a single direction and a bad reward?

---

Edit:

• How should an agent with action size more than one learn about a good state?
• Reward attribution in deep q learning and texas holdem poker
• Is experience replay taking individual memory samples out of context? have partly answered my question. However if anyone still wants to answer my question, I would still appreciate it.

Answer 1

I'm not sure this is a big deal. For one thing, the agent isn't maximizing one-step reward but future return (sum of future reward). So in that sense, the agent in your example will attribute a higher weight to the earlier decision of moving right a lot.

If you consider the state-action-reward sequence $\{s_1, \text{right}, 1, s_2, \text{right}, 1, s_3, \text{left}, 1, s_4, \text{right}, 0, s_\text{terminal}\}$ and (for simplicity) the REINFORCE objective then we have that

$$\nabla_\theta J(\theta) = 3 \cdot \nabla_\theta \log \pi_\theta (\text{right} | s_1) + 2 \cdot \nabla_\theta \log \pi_\theta (\text{right} | s_2) + 1 \cdot \nabla_\theta \log \pi_\theta (\text{left} | s_3) + 0 \cdot \nabla_\theta \log \pi_\theta (\text{right} | s_\text{terminal}).$$

Now imagine if the last action was left, ending in termination. Then that term would be multiplied by $0$ and rather inconsequential. Likewise, if it was able to recover and extend the episode by another 10 time steps, the return would be higher and that sequence of actions would be good overall. In this set-up, I do not see the point of delaying reward since the return (or value) is what is being optimized.

For something more complicated like DQN, it is still true that return is being optimized - not immediate reward. Furthermore, it's not correct to say that the agent will memorize certain actions in certain states. That is only true in the tabular setting. This is also why we require that the policy is suitably exploratory. That way, the agent won't get "stuck" in this behavior loop.