I am trying to solve the snake game below with a DQN agent.
Actually I was originally trying to solve a much larger grid problem, however have made it simple to essentially an 8x8 RGB state representation of the environment, as I've had little luck.
However, my snake still doesn't seem to learn to head towards the blue foods in order to get a reward (+1). It mostly just twirls around a bit, partly due to an annealing epsilon of 2, before hitting the edge to end the episode (with -1 reward).
I have the following CNN to help with determining action values:
self.conv1 = nn.Conv2d(3, 8, 2, 1) self.relu1 = nn.ReLU(inplace=True) self.conv3 = nn.Conv2d(8, 8, 1, 1) self.relu3 = nn.ReLU(inplace=True) self.fc4 = nn.Linear(392, 200) self.relu4 = nn.ReLU(inplace=True) self.fc5 = nn.Linear(200, 80) self.relu5 = nn.ReLU(inplace=True) self.fc6 = nn.Linear(80, self.number_of_actions) self.softmax = nn.Softmax()
Although this state representation is unlikely to be the issue for such a simple 8x8 environment, and I get the same issue when simply unrolling the pixels into a full connected deep neural network.
I do see that my agent is learning something as it often tries to mimic previous series of actions that luckily led to the consumption of food.
However a mere total of 0 (+1, -1) rewards is as good as it gets, with most episodes terminating with -1 rewards.
I am using a replay buffer which generously samples 100 times to update the DQN network. I also training for around 1,000 episodes with no success.
Any suggestions? Thanks.