Loss not decreasing but performance is improving

Question

I am training a custom implementation of DQN on the SpaceInvaders environment from OpenAI gym. The episode reward keeps increasing and approximately reaches the maximum episode reward that DQN achieves in this environment. However, the strangest thing is that initially the loss increases and then it keeps rapidly oscillating without showing any consistent decrease?

How can this be explained? My first guess was the moving target in DQN but in other games (e.g. Pong), a smooth gradual decrease can be observed? It is kind of clear that a decrease in the learning rate would probably fix the oscillation. However, I am interested in an explanation for the specific results in the image, and specifically how does it learn if the loss is not decreasing.

NOTE: The environment is definitely the same as the one used in the original DQN paper. The loss plotted below is the L2 loss, WITHOUT the huber loss applied on top of it as in the original DQN paper. Instead, all gradients are truncated to norm 10.

Answer 1

This is not unusual for reinforcement learning and does not indicate anything is wrong. As the agent gets better at playing, estimating the reward does get more difficult (because it's no longer always 0). In addition, as the reward gets higher, and the average episode length gets longer, the amount of variance in the reward can also get larger, so it's challenging even to prevent the loss from increasing. A third factor, which you mentioned, is that the constantly changing poses a "moving-target" problem for the Q-network.

My guess for why this doesn't happen in Pong is that Pong is a much simpler game and easier to predict than Space Invaders.