Why do we need to exploit when training an agent in reinforcement learning?

Question

I have been studying reinforcement learning lately for a robotic project but I have been confused about one thing in particular.

When the agent is learning during the training phase, why is it important that the agent exploits rather than explores? My idea is that since it's learning, it should be free to explore different policies and not restrict it to any degree of exploitation. This way, each step will certainly be investigated and every combination of steps in an episode will be traversed. Isn't it better this way or am I missing an important piece of the puzzle?

Answer 1

A conceptual answer might be that exploitation reveals interesting areas for exploration. For example, if you try to learn to write a story by typing random words, you likely won't get far, no matter how much you try. On the other hand, once you've "exploited" and mastered how to write grammatical sentences, structure an interesting plot, use the standard character tropes -- then it's much easier to explore different styles and forms of writing.

Theoretically, it's possible off-policy algorithms such as Q-learning to have an rollout policy which is totally different and unrelated from the greedy policy. However, in practice, if the states encountered in the greedy policy are very far from those encountered in the rollouts, the Q-function approximator will probably not generalize well.

As for on-policy algorithms, by definition, rollouts need to be sampled from the current policy, so there is no way to improve the policy using arbitrary explorative rollouts.