0
$\begingroup$

I am using deep q learning (output layer activation function is tanh) for a reinforcement learning agent to navigate in a grid. What it learns is not to hit the walls and other obstacles. If it hits something it get -0.9 reward and 0 otherwise. After sometime the agent begins to go one or two steps forward and then backwards and again forward and so on. To break out from this oscillation I can use exploration (10% or 1% or something). But if i do that it remedy the oscillation problem but it affect the actual functionality since now the agent will take a random action once every ten times and thus will hit something even though the system has learned not to. Usually how do people address such an issue ? Thank you

$\endgroup$
1
$\begingroup$

In your question you explain the following rewards:

  • -0.9 for hitting a wall or obstacle

  • 0 otherwise

Your agent's behaviour is:

to go one or two steps forward and then backwards and again forward and so on

With the information you have given, this appears to be successful. The agent has learned to optimally solve the problem you have set it, as defined by the rewards.

It is not completely clear what the rest of the environment is like, what the agent "observes", and what behaviour you would like to train the agent to achieve. However, you should start by looking at the reward structure. Reinforcement learning is all about the agent accumulating the highest total reward.

A couple of suggestions:

  • If you want the agent to navigate to a specific position quickly, then make the problem episodic - ending when it reaches the goal - and add a small negative reward for each step (maybe -0.1). Maybe time limit the episode and grant an end reward for how close the agent is to the goal.

  • If you want the agent to explore the space, the you need to have a reward signal for what counts as exploration. For example, grant +0.1 reward if the agent moves to a position it has not been in before. If this is not an episodic problem, maybe let the rewards from visited squares grow back up over time to some maximum.

Unrelated to reward, if you want to simulate a more complex motion, such as control in driving, you may need to switch away from a grid world where the agent has free choice of movement and absolute control, and use a simplified physics, giving the agent imperfect control over steering. This takes a lot more effort to set up the environment, but can result in more complex behaviours.

$\endgroup$
  • 1
    $\begingroup$ @EshanM.Herath: One thing I missed - you probably need to lose the $\text{tanh}$ activation function in the output layer. With some exceptions, when you are calculating Q (the action value) you need your estimator to be able to predict higher and lower returns, whilst $\text{tanh}$ is limited to $[-1,1]$ $\endgroup$ – Neil Slater Oct 14 '17 at 6:47
  • $\begingroup$ Thank you. And does that mean I should use a linear activation function at the output layer. If so do i have to define a ceiling and a floor for the output values. $\endgroup$ – user112485 Oct 14 '17 at 11:28
  • 1
    $\begingroup$ @EshanM.Herath: Yes linear activation function and MSE loss are good general choices, although you could also scale the maximum possible returns (not rewards) so everything fits into $[-1,1]$. You don't have to define floors/ceilings as far as I know, but NNs in RL can be unstable, so it might be something that will help. $\endgroup$ – Neil Slater Oct 14 '17 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy