1
$\begingroup$

I am applying DQ Learning to a continuous action space with rewards received at the end of each trial. My agent is in a fixed 24step long setting where it receives the reward at the end of those 24 steps.

DQ Learning uses experience replay and randomly samples out of these areas. That would mean that if I placed all these actions into my replay buffer, the agent would learn 23/24 times that whatever action it takes, the reward is 0. And then it would associate whatever action it took in the last of the 24 steps as the rewarding action. However, that is not at all what I want to learn.

So, can it be that the DQN learning approach dissolved the idea of the agent being successfully considering "expected future reward" in its policy updates? Or did I misinterpret DQN learning?

I read the typical papers on this:

All of those play Atari games and I believe they continuously receive a reward, so they can just randomly sample an state-action-rew-state tuple because they get a reward for almost all actions (more or less).

$\endgroup$
2
$\begingroup$

the agent would learn 23/24 times that whatever action it takes, the reward is 0

The Q-function maps a state-action pair to the expected discounted cumulative future reward, not the immediate reward of taking an action. Therefore it's perfectly fine to run Q-learning with sparse rewards. (It can be problematic for other reasons though).

$\endgroup$
  • $\begingroup$ how can the agent expect a future reward, if the transition has never been observed together with the following transitions that actually lead to the reward? How does the expected future reward get passed form the final step to the first step if experience replay doesn't sample across whole trajectories but just across single S-A-R-S tuples? $\endgroup$ – pascalwhoop Jun 8 '18 at 18:38
  • $\begingroup$ @pascalwhoop the reward gets "passed" backwards due to the way the Q-function is recursively defined using future states. Look at the loss function for Q-learning. $\endgroup$ – shimao Jun 9 '18 at 2:15
  • $\begingroup$ @pascalwhoop: Initially, when the agent runs its first few episodes, your intuition that the agent cannot possibly know is correct. In fact this is one of the factors related to problems with stability of Q-learning algorithm and neural networks. However, once the agent learns where to find at least some better immediate rewards and the transitions that lead to them, using them in experience replay, then the updates to the estimate for Q will do as shimao says and feed back to improved estimates of future reward in earlier states. $\endgroup$ – Neil Slater Jun 9 '18 at 7:49
  • $\begingroup$ @NeilSlater I could adapt the reward function a bit to give it a reward also "on the way" I believe that will largely outperform the alternative approach. $\endgroup$ – pascalwhoop Jun 9 '18 at 10:24
  • $\begingroup$ @pascalwhoop: I would advise against meddling with the reward scheme. Essentially by doing this you are inserting your own "expert" heuristics or prior knowledge for the agent. There may be better approaches for sparse reward schemes - e.g. prioritised sweeping for planning, or TD($\lambda$) to update more of each trajectory at once in Q learning. Although it depends what your purpose is? $\endgroup$ – Neil Slater Jun 9 '18 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.