I have a reinforcement learning environment where the agent has to reach a certain target position by moving its limbs. At the moment, I stop each episode after 2000 simulation steps. During the first 100 episodes or so, the target is rarely reached. But after a while, the target is often reached.

Should I stop the simulation once the target has been reached?

To me, it seems like this question is the balance between two aspects:

  • Running the simulation for too long will flood the buffer (i'm using experience replay) with samples that are not too relevant for actually getting the agent to the target position.
  • However, these samples are necessary to show that there is a very large long-term reward when reaching to the target.

So what's the consensus?

  • $\begingroup$ What is the goal of this agent? Is it really to stay in position at the end (is anything preventing that?) or is it to reach the end point as fast as possible? Do you need it to reach the end point and stop moving (i.e. reaching the end point and having a large velocity is not a good end state)? $\endgroup$ Feb 3 at 20:06
  • $\begingroup$ @NeilSlater nothing is preventing it to stay in place. Yes, it has to reach the end point and then stop moving, so in that sense it's wise not to stop the episode directly. However, I could also implement a secondary reward that minimizes the velocity $\endgroup$ Feb 4 at 8:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.