0
$\begingroup$

enter image description here

Specifically, for a deep deterministic policy gradient, DDPG, to expedite the learning speed, it's recommended to use a Replay Buffer

What if the reward is only given at a terminal state? Or, most of the experiences rarely give immediate rewards but the reward at the terminal state is sort of "what really matters"?

Cause in that case, in that sample of a random minibatch, if it does not include the terminal state, then you would most likely have many 0 immediate rewards. Would DDPG algorithm with Replay Buffer still work for such situation?

$\endgroup$
1
$\begingroup$

Cause in that case, in that sample of a random minibatch, if it does not include the terminal state, then you would most likely have many 0 immediate rewards. Would DDPG algorithm with Replay Buffer still work for such situation?

Yes it will work just fine. The point of replay buffer is not to find non-zero rewards, it stores observed state transitions, and these are critical to resolving the credit assignment problem. Every time the algorithm process a transition with zero reward, it also processes the link between state, action and next state. This link between states is critical to picking correct actions when rewards are sparse.

If all the experience only has zero rewards, then of course there is nothing useful to learn yet (the agent may as well take random actions as far as it knows). Also, environments with very sparse rewards are harder to learn.

However, there is no problem in general if a minibatch contains only samples with zero reward. The agent can still learn just fine in that case, as the learning targets are not just based on reward on the single step, but also predictions of future reward.

Biased sampling towards non-zero rewards might help in some cases, but a more robust related approach is to bias towards action values where there was a large change recently to next state's action values - that is the premise of prioritised sweeping.

$\endgroup$
  • $\begingroup$ But would it affect overall training time in practice? Cause intuitively speaking, I would assume it would... $\endgroup$ – user355843 Feb 16 at 12:36
  • $\begingroup$ @user355843: Yes it might, but focusing on just the last parts of a trajectory regardless of progress is probably too naive to use in general. Whilst prioritised sweeping takes the idea further and is more general. $\endgroup$ – Neil Slater Feb 16 at 13:18

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.