Reinforcement Learning - what causes a plateau in the performance of the model during training?

Question

I am working on a Reinforcement Learning problem, but since the underlying model is a neural network, I think this might have similarities to a supervised learning problem.

Below is a screenshot of my model's mean performance over 5 million training steps:

You can see that around 1-1.5M steps, it reaches a certain level of performance and just oscillates around that level for the rest of its training. I'm wondering, what can cause this behavior? I thought that if I keep training a neural network, it will keep overfitting more and more to my data.

I realize that Reinforcement Learning is somewhat different, but my RL environment comes from a tabular dataset, so there's not an infinite amount of data/randomness/stochasticity. Why would the performance plateau like this?

From knowing the data, I know for certain that it is possible to achieve a higher score on the data (maybe not on a test set, but for a model fitting itself to the training set, absolutely).

Answer 1

I think I figured it out. Inspired by this post, I basically introduced a learning rate schedule.

Essentially, the problem is that the model reaches a point where it needs more fine-tuned changes, but because my learning rate was too high for that part of the training, the update steps the model was taking were too large, and it was unable to smoothly converge to the right trainable parameters.

Over the course of training, I decreased the learning rate. So I started with a larger learning rate (to take initial learning steps faster), and then around the time of the plateau, I dropped the learning rate so that the model could do more fine-tuning. This removed the plateau and allowed the model to continue getting higher scores.