I have a reinforcement learning environment where the agent has to reach a certain target position by moving its limbs. At the moment, I stop each episode after 2000 simulation steps. During the first 100 episodes or so, the target is rarely reached. But after a while, the target is often reached.
Should I stop the simulation once the target has been reached?
To me, it seems like this question is the balance between two aspects:
- Running the simulation for too long will flood the buffer (i'm using experience replay) with samples that are not too relevant for actually getting the agent to the target position.
- However, these samples are necessary to show that there is a very large long-term reward when reaching to the target.
So what's the consensus?