Assume we want to train an agent where the agent should select a vector of K actions (the action size for this agent is K). More specifically, assume a grid environment where we have K robots where each robots can move up, down, left, right. There are K specific goal locations where the robots should go there. In each episode, if each robot occupy one goal (so that all goal locations are occupied), we get the reward of 1 and this episode ends; otherwise, for each time passed we get -0.01 (a punishment so our agent learns to do this task faster). Each episode have T time steps and the agent receives the sum of rewards obtained at each time step. I want to use a DQN to train this agent.
What we are usually do when using a DQN is that we fill the replay memory of size M with random actions (without updating the network). When the memory is full, we start taking batch from memory and update our network while following an epsilon greedy algorithm (decreasing epsilon from 1 to 0.05 in L time steps).
The issue here is that the probability of observing the positive reward for our agent is almost zero. So, it is quite possible that when we fill the memory, all experiences have the episode reward of -0.01*T. Even it is quite possible that all L exploration steps end and the agent does not observe the reward of 1.
Is there any way to remedy this issue? This issue become more severe as we increase K.
Is any of following ways is going to help?
- Increase T to a very large number so that we make sure that the experience regarding positive reward happens.
- Increase L to increase the chance of experiencing positive reward happens. In this case, we need to increase M proportionally.
Obviously, if we give partial positive rewards to the agent, it can learn it easier (and faster).