I am applying DQ Learning to a continuous action space with rewards received at the end of each trial. My agent is in a fixed 24step long setting where it receives the reward at the end of those 24 steps.
DQ Learning uses experience replay and randomly samples out of these areas. That would mean that if I placed all these actions into my replay buffer, the agent would learn 23/24 times that whatever action it takes, the reward is 0. And then it would associate whatever action it took in the last of the 24 steps as the rewarding action. However, that is not at all what I want to learn.
So, can it be that the DQN learning approach dissolved the idea of the agent being successfully considering "expected future reward" in its policy updates? Or did I misinterpret DQN learning?
I read the typical papers on this:
- Original DQ by DeepMind on Atari https://arxiv.org/abs/1312.5602
- Continuous DeepQ with NAF http://proceedings.mlr.press/v48/gu16.pdf
- Continuous control with DQL: https://arxiv.org/abs/1509.02971
All of those play Atari games and I believe they continuously receive a reward, so they can just randomly sample an state-action-rew-state tuple because they get a reward for almost all actions (more or less).