1
$\begingroup$

I am trying to solve the snake game below with a DQN agent.

Actually I was originally trying to solve a much larger grid problem, however have made it simple to essentially an 8x8 RGB state representation of the environment, as I've had little luck.

However, my snake still doesn't seem to learn to head towards the blue foods in order to get a reward (+1). It mostly just twirls around a bit, partly due to an annealing epsilon of 2, before hitting the edge to end the episode (with -1 reward).

I have the following CNN to help with determining action values:

self.conv1 = nn.Conv2d(3, 8, 2, 1)
self.relu1 = nn.ReLU(inplace=True)
self.conv3 = nn.Conv2d(8, 8, 1, 1)
self.relu3 = nn.ReLU(inplace=True)
self.fc4 = nn.Linear(392, 200)
self.relu4 = nn.ReLU(inplace=True)
self.fc5 = nn.Linear(200, 80)
self.relu5 = nn.ReLU(inplace=True)
self.fc6 = nn.Linear(80, self.number_of_actions) 
self.softmax = nn.Softmax()

Although this state representation is unlikely to be the issue for such a simple 8x8 environment, and I get the same issue when simply unrolling the pixels into a full connected deep neural network.

I do see that my agent is learning something as it often tries to mimic previous series of actions that luckily led to the consumption of food.

However a mere total of 0 (+1, -1) rewards is as good as it gets, with most episodes terminating with -1 rewards.

I am using a replay buffer which generously samples 100 times to update the DQN network. I also training for around 1,000 episodes with no success.

Any suggestions? Thanks.

enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ Your question makes me wanna relearn python because I do not get your code and can't see what went wrong. ------ What I imagine went wrong is that you have not been training your snake well, by not giving sufficient input (how do you expect your snake to learn?).... $\endgroup$
    – Sextus Empiricus
    May 8, 2020 at 10:17
  • $\begingroup$ ...A good snake should be able to cover the entire field without hitting the boundary (If it knows it's present coordinate, does your snake knows it's own coordinate? What is the background of the game?), even without any potential reward of food present. This makes me think of my simplistic battleship game strategy which ignores history and is just boringly hitting the field according to a regular meshgrid pattern. $\endgroup$
    – Sextus Empiricus
    May 8, 2020 at 10:17
  • $\begingroup$ Hi , is there any problem still going on? @noob $\endgroup$
    – Emir Ceyani
    May 13, 2020 at 15:05
  • $\begingroup$ @EmirCeyani Unfortunately yes. The issue was not the state representation itself or the fact that a softmax was used. Perhaps it requires more training time. I put the project on hold, but will continue next week. I will look further into the Dueling DQN Architecture mentioned. Thank you very much for that :) $\endgroup$
    – noob
    May 24, 2020 at 9:29
  • $\begingroup$ @SextusEmpiricus Many thanks for your response. Mind you the code provided is part of a CNN written with PyTorch. Indeed the Q-learning would consider past history for the state-action space. Perhaps the issue is, as you said may be insufficient inputs... in this case likely due to the sparse rewards scenario and perhaps also the very challenging RGB state representation of the environment provided. $\endgroup$
    – noob
    May 24, 2020 at 9:32

1 Answer 1

Reset to default
2
+100
$\begingroup$

The problem with your DQN architecture is you are using softmax output which limits the possible values of your Q matrix no matter how longer you train. Given a state , $s$, you should return a vector of real numbers for all possible actions and train your model with smooth L1(Huber) or L2 loss as learnign Q values is estimation/ regression problem . Softmax can be used for representing the policy network/function for discrete actions. However for Deep Q learning, we are learning an approximate estimate of the Q value for each state action pair, which is a regression problem. Softmax hampers the possible $Q(s,a)$ values.

You can also check Dueling DQN Architecture for a better improvement.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.