1
$\begingroup$

I have been studying reinforcement learning lately for a robotic project but I have been confused about one thing in particular.

When the agent is learning during the training phase, why is it important that the agent exploits rather than explores? My idea is that since it's learning, it should be free to explore different policies and not restrict it to any degree of exploitation. This way, each step will certainly be investigated and every combination of steps in an episode will be traversed. Isn't it better this way or am I missing an important piece of the puzzle?

$\endgroup$

1 Answer 1

1
$\begingroup$

A conceptual answer might be that exploitation reveals interesting areas for exploration. For example, if you try to learn to write a story by typing random words, you likely won't get far, no matter how much you try. On the other hand, once you've "exploited" and mastered how to write grammatical sentences, structure an interesting plot, use the standard character tropes -- then it's much easier to explore different styles and forms of writing.

Theoretically, it's possible off-policy algorithms such as Q-learning to have an rollout policy which is totally different and unrelated from the greedy policy. However, in practice, if the states encountered in the greedy policy are very far from those encountered in the rollouts, the Q-function approximator will probably not generalize well.

As for on-policy algorithms, by definition, rollouts need to be sampled from the current policy, so there is no way to improve the policy using arbitrary explorative rollouts.

$\endgroup$
1
  • $\begingroup$ Thank you for the nice analogy. So, is it correct to conclude that we exploit because of the limited computational resources (e.g. hardware and time) while we explore to discover better policies? And is this what the exploit-explore tradeoff is all about in RL? $\endgroup$ Jun 23, 2021 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.