I'm currently working on a project about Reinforcement Learning in game playing, and I need to first do experiments of DQN. I tried to implement a version for Atari Boxing game by changing the code in http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html.
I did an experiment with hyper-parameters of the original paper scaled by 0.01. To be specific:
- the memory size is now 10000
- the frequency for updating the target network is every 100 training steps
- the number of steps to decrease the epsilon is 10000
- the number of training steps is 500000, which is divided into 500 training episodes with 1000 episode length
- the evaluation episode length is 500000
I also used some functions different from the original paper:
- Huber Loss for loss
- an exponential function for decaying epsilon
- populating to fill all of the memory
- pretrain the network so with next state value 0 (similar to Q-learning), as in training I don't distinguish between terminal states and nonterminal states
- no random start, and no repeat of actions
The first picture shows the cumulative reward every 100 frames, the second shows the average loss during 100 frames, and the final one shows the evaluated reward for 30 trials.
It seems that the policy was not improving at all. I don't know what I should work on to deal with this problem. Could you offer me some suggestions?