I am using deep q learning (output layer activation function is tanh) for a reinforcement learning agent to navigate in a grid. What it learns is not to hit the walls and other obstacles. If it hits something it get -0.9 reward and 0 otherwise. After sometime the agent begins to go one or two steps forward and then backwards and again forward and so on. To break out from this oscillation I can use exploration (10% or 1% or something). But if i do that it remedy the oscillation problem but it affect the actual functionality since now the agent will take a random action once every ten times and thus will hit something even though the system has learned not to. Usually how do people address such an issue ? Thank you
In your question you explain the following rewards:
-0.9 for hitting a wall or obstacle
Your agent's behaviour is:
to go one or two steps forward and then backwards and again forward and so on
With the information you have given, this appears to be successful. The agent has learned to optimally solve the problem you have set it, as defined by the rewards.
It is not completely clear what the rest of the environment is like, what the agent "observes", and what behaviour you would like to train the agent to achieve. However, you should start by looking at the reward structure. Reinforcement learning is all about the agent accumulating the highest total reward.
A couple of suggestions:
If you want the agent to navigate to a specific position quickly, then make the problem episodic - ending when it reaches the goal - and add a small negative reward for each step (maybe -0.1). Maybe time limit the episode and grant an end reward for how close the agent is to the goal.
If you want the agent to explore the space, the you need to have a reward signal for what counts as exploration. For example, grant +0.1 reward if the agent moves to a position it has not been in before. If this is not an episodic problem, maybe let the rewards from visited squares grow back up over time to some maximum.
Unrelated to reward, if you want to simulate a more complex motion, such as control in driving, you may need to switch away from a grid world where the agent has free choice of movement and absolute control, and use a simplified physics, giving the agent imperfect control over steering. This takes a lot more effort to set up the environment, but can result in more complex behaviours.