In DeepMind's paper on Deep Q-Learning for Atari video games (here), they use an epsilon-greedy method for exploration during training. This means that when an action is selected in training, it is either chosen as the action with the highest q-value, or a random action. Choosing between these two is random and based on the value of epsilon, and epsilon is annealed during training such that initially, lots of random actions are taken (exploration), but as training progresses, lots of actions with the maximum q-values are taken (exploitation).
Then, during testing, they also use this epsilon-greedy method, but with epsilon at a very low value, such that there is a strong bias towards exploitation over exploration, favouring choosing the action with the highest q-value over a random action. However, random actions are still sometimes chosen (5 % of the time).
My questions are:
Why is any exploration necessary at all at this point, given that training has already been done?
If the system has learned the optimal policy, then why can't the action always be chosen as the one with the highest q-value?
Shouldn't exploration be done only in training, and then once the optimal policy is learned, the agent can just repeatedly choose the optimal action?