1
$\begingroup$

I am trying to replicate the results of DeepMind's DQN paper (https://arxiv.org/pdf/1312.5602v1.pdf) for the particular case of Atari Pong.

However, I am observing a (maybe not so) weird behavior: after a few game episodes (say, 5 or 6) my Q-network starts outputting zeros regardless of the input state!

Initially, I thought this was due to a bug in my code, but after thinking for a while I realized that it somehow makes sense. The rewards $\lbrace r_t \rbrace$ are zero for the vast majority of the states $\lbrace s_t \rbrace$ (except for those where we score or concede a goal, getting a $+1$ or $-1$ reward, respectively) and the Bellman equation is:

$Q(s_t, a_t) = r_t + \gamma \max_a Q(s_{t+1}, a)$

Thus, for $r_t=0$, the Bellman equation is satisfied (i.e., the loss is zero) if $Q(s,a)=0 \,\, \forall s,a$. Because zero rewards happen very often in Pong, it makes sense that the model is converging towards that solution... Any hints on how I can solve this?

Remark 1: I think I do not have any bug in the Q-learning algorithm, since I could solve the Cart-Pole problem using the same code (with different Q-network and hyperparameters)

Remark 2: I am using the network architecture and image preprocessing that were presented in the original paper. I have also implemented some extra details described in the link below, which I found while searching for a solution in this forum.

https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/

$\endgroup$
  • $\begingroup$ After 5 or 6 episodes does not seem very long for this problem. Typically the Atari emulator and agent are left running for hours running games in an accelerated environment. $\endgroup$ – Neil Slater Nov 28 '17 at 9:42
  • $\begingroup$ True, but that's when the function starts being zero. After that, it simply gets stuck at zero (I have run more than 5k episodes right now). $\endgroup$ – D... Nov 28 '17 at 10:32
  • $\begingroup$ Not seeing your code, I would suspect a bug such as finishing the episode before the reward-gaining step is being stored in the history table (e.g. some logic which detects the end of the episode is running before your "save experience" call). Take a look in the history table and see if any S, A, R, S' record has a non-zero R. $\endgroup$ – Neil Slater Nov 28 '17 at 11:15
  • $\begingroup$ Nevermind, there's actually a very silly bug in my code: I was using a ReLU at the output layer XD $\endgroup$ – D... Nov 28 '17 at 11:56
  • $\begingroup$ OK, yes that would definitely cause some problems with negative rewards. I guess your cartpole implementation had +ve reward for each timestep . . . $\endgroup$ – Neil Slater Nov 28 '17 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.