3
$\begingroup$

I have a simple grid environment. The player is in the upper left corner and it's goal is to get to lower right corner. The player receives +0.2 points for moving in the direction of the goal, -0.2 for moving in the opposite direction and -0.001 time penalty every step. It also scores +1 for winning and -1 for dying. Now there are two settings. One with two enemies who move randomly on the map and one without them. My network learns how to play the latter setting in a matter of a few games but seem to be unable to consistently win in the first setting. I tried tweaking hyperparameters but to no avail. Cumulative reward

As you can see the agent actually manages to win a few games at first but seems to forget this ability later on. The agent gets about 1 point after some training because all it learnt is to go right even after hitting the edge of the map and wait for death there. All the variance comes from the fact that it takes a random number of steps for the enemies to get to the player.

Loss

Low loss suggests that the predictions made for the policy are correct, but the policy itself is not improving.

I've tried two neural networks - A large, deep CNN with the entire map as input and a small fully connected one that takes coordinates of player and two enemies as input and has two hidden layers with 100 neurons each. Both networks get comparably bad results.

I'm using adam optimiser, learning rate = 3e-4, weight decay = 0.05, discount factor = 0.9

I believe there are no mistakes in the Q-learning algorithm because it has been tested on a few gym games and the setting without the enemies and the agent does learn to win these games.

What are the possible reasons for this behaviour and how can I fix it?

$\endgroup$
1
  • $\begingroup$ Unfortunately I didn't manage to find the mistake in the code. I ended up rewriting the entire algorithm from scratch using another framework and it did help. So it seems it actually was a mistake in the algorithm but I don't know where. :/ $\endgroup$
    – YuseqYaseq
    Commented Jan 19, 2020 at 15:54

1 Answer 1

1
$\begingroup$

One possible explanation might be a result of the game setting.
Regardless of the cell the player is in - if the probability($P$) of it winning equals(or less) the probability of it meeting an enemy(and losing) then its best move is moving right.
$P$ depends on board's size and enemies initial positions.
The smaller the board the higher the probability of an encountering an enemy.

I guess the player 'forget its ability' after randomly losing a few a games and learning the 'true' expected value of moving down.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.