An agent receives an extrinsic reward $r_{ext}$ and an intrinsic reward $r_{int}$ and a Q-function approximation is trained using TD learning such that $Q(s,a)$ approximates the expected return of $r_{ext} + \beta r_{int}$ where $\beta$ is a coefficient less than 1.

In online reinforcement learning, what is a good policy to use when there is intrinsic rewards? For example, if $\epsilon$-greedy is used, then there is a probability of $\epsilon$ that a random action is selected (i.e., undirected exploration). This ignores the intrinsic reward which is a form of directed exploration. If a greedy policy is used, then this risks myopic exploration within a subset of the state space and the agent might never experience certain intrinsic rewards that guides it to meaningful states. It seems then that $\epsilon$ needs to be tune correctly: too high and it ignores intrinsic rewards, too low and it is myopic.

On the other hand, are there other policies that can be used? Even if $\epsilon$ is small, say $0.2$, the agent still ignores directed exploration 2 out of 10 times.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy