Overfitting, and generalization, are quite different in reinforcement learning than in supervised learning. There's perhaps a joke to be made that statisticians fit to the training set, machine learning people fit to the test set and reinforcement learning people overfit to the test set. However, this is not quite fair.
In RL, you are trying to approximate the solution to a Markov Decision Process (MDP). When you are doing this with a simulator, you have as much interaction with it as you would like - you can always collect more data. Then, how do you delineate between a "training" and "testing" phase? Furthermore, why would you at all? I would argue that with a simulator, you shouldn't care about the difference between "training" and "testing" because you can evaluate your model through policy evaluation - a central topic in RL.
The story is slightly different in two situations: when you want to learn over a class of MDPs or when you are using RL "in the real world" (read: without a simulator). In the first case, this is an instance of transfer learning. You want to take a policy in one MDP and have it generalize to another MDP. There is current work on this (https://arxiv.org/pdf/1812.02868.pdf, https://arxiv.org/pdf/1810.12282.pdf, https://arxiv.org/pdf/1812.02341.pdf and many others). In the second case, you can do policy evaluation on a hold out set of data similar to supervised learning (see work done by Emma Brunskill on this).
One last note on the subtleties of testing in supervised vs reinforcement learning. It's not quite fair to fix an RL agent and evaluate it. This gives an advantage to supervised learning approaches, since they are optimized to succeed in this fix-and-evaluate regmine. RL places fundamental importance on online learning, and so an RL agent should be evaluated in an online learning regime.
