# Playing Atari: Q-function converges to zero!

I am trying to replicate the results of DeepMind's DQN paper (https://arxiv.org/pdf/1312.5602v1.pdf) for the particular case of Atari Pong.

However, I am observing a (maybe not so) weird behavior: after a few game episodes (say, 5 or 6) my Q-network starts outputting zeros regardless of the input state!

Initially, I thought this was due to a bug in my code, but after thinking for a while I realized that it somehow makes sense. The rewards $\lbrace r_t \rbrace$ are zero for the vast majority of the states $\lbrace s_t \rbrace$ (except for those where we score or concede a goal, getting a $+1$ or $-1$ reward, respectively) and the Bellman equation is:

$Q(s_t, a_t) = r_t + \gamma \max_a Q(s_{t+1}, a)$

Thus, for $r_t=0$, the Bellman equation is satisfied (i.e., the loss is zero) if $Q(s,a)=0 \,\, \forall s,a$. Because zero rewards happen very often in Pong, it makes sense that the model is converging towards that solution... Any hints on how I can solve this?

Remark 1: I think I do not have any bug in the Q-learning algorithm, since I could solve the Cart-Pole problem using the same code (with different Q-network and hyperparameters)

Remark 2: I am using the network architecture and image preprocessing that were presented in the original paper. I have also implemented some extra details described in the link below, which I found while searching for a solution in this forum.

https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/

• After 5 or 6 episodes does not seem very long for this problem. Typically the Atari emulator and agent are left running for hours running games in an accelerated environment. – Neil Slater Nov 28 '17 at 9:42
• True, but that's when the function starts being zero. After that, it simply gets stuck at zero (I have run more than 5k episodes right now). – D... Nov 28 '17 at 10:32
• Not seeing your code, I would suspect a bug such as finishing the episode before the reward-gaining step is being stored in the history table (e.g. some logic which detects the end of the episode is running before your "save experience" call). Take a look in the history table and see if any S, A, R, S' record has a non-zero R. – Neil Slater Nov 28 '17 at 11:15
• Nevermind, there's actually a very silly bug in my code: I was using a ReLU at the output layer XD – D... Nov 28 '17 at 11:56
• OK, yes that would definitely cause some problems with negative rewards. I guess your cartpole implementation had +ve reward for each timestep . . . – Neil Slater Nov 28 '17 at 12:19