Understanding training phase of an agent trained using deep Q-learning Assume we have a deterministic game and we want to train an agent using Deep Q-Learning (a.k.a. DQN) to play this game. We record the score of the agent at each episode and plot the scores versus the episode numbers. Assume the maximum score of the game is 1 and the minimum is -1.
Now, my question is this: if we trained an agent for 300k episodes and observed after a while the agent learns to get the score of 1 and it keeps doing that for the first 100k. However, after that, the score drops to around -1 and stays there until 300k-th episode. What can be the reason for this phenomena? Although the agent learns to play the game perfectly for a long time, why it suddenly forgets it?
 A: This is called "catastrophic forgetting", and seems to occur quite frequently in DQN.
The problem you are seeing might go something like this internally (note this is a hand-wavy/conjecture argument, I have not measured thi happening numerically):


*

*The agent learns a near-optimal strategy. 

*It follows that strategy, and only explores close to it. 

*The transition history becomes saturated with states and actions seen close to optimal behaviour, and that more often than not gain long-term rewards of optimal behaviour.

*The model parameters, in order to obtain the best loss for those states, change so that they predict action values as accurately as possible for the current distribution of states and actions.

*The accuracy of predictions for less-frequently seen states/actions gets worse due to over-fitting. Initially this is not noticed, because those state/action pairs are not being visited.

*Eventually, a prediction for a bad action choice somewhere in the environment becomes high enough to get that action selected, perhaps whilst exploring. This makes the agent explore a "bad neighbourhood" of states and actions with low rewards.

*The neural network tries to correct the predictions whilst taking these bad actions, but now the internal representations that were focused on the optimal path only work against it - any correction to this erroneously high prediction for a bad action will at least initially change parameters so that the values of all optimal state/actions are also reduced.

*Depending on how far the over-fitting went before the agent made a poor enough prediction that made it misstep, it could either recover relatively quickly, or take a long time. The high error signals comparing predicted value against actual value could even cause it to diverge.
Some fixes to this are the same as for over-fitting - add regularisation to the neural network, test your progress (use monte-carlo estimates of the agent following the Q values perfectly over some number of episodes).
You might also be tempted to adjust what is stored in replay memory. It may help if you find a simple heuristic for this, or perhaps you are able to start the agent in a variety of random starting states. However, in general this is a really hard unsolved problem - the statistical dataset/population base assumed for supervised ML is not a perfect match for learning Q-functions, which don't have quite the same relationship to populations of observed transitions - at least not while exploring and learning optimal/better control.
There is ongoing research into this problem, with various proposed fixes, such as Overcoming Catastrophic Forgetting by Incremental Moment Matching.
