I'm trying to understand exactly how to use experience replay in DQN but I'm not sure I understand how it's done.

Here's how I think it develops:

Observe state s
Take action a
Observe reward r and new state s'
Store transition in replay memory D(s,a,r,s')

Now my question is where exactly do we go from here? do we train the network and update the weights here, then on the next transition, instead of training the network on that transition, we sample a random one from the replay memory and update the weights like below?

Update weights
take action a
observe reward r and new state s'
Store new transition in D
sample random transition from D (which till now has only 2 entries)
Update weights according to sampled transition

Is this how it's done?

  • $\begingroup$ Could you spell out DQN? $\endgroup$ – kjetil b halvorsen Nov 29 '16 at 14:07
  • $\begingroup$ @kjetilbhalvorsen If I do, will you help? :D $\endgroup$ – naimelhajj Nov 29 '16 at 14:12

What you are describing at the end is online learning, where we are continually updating the approximation of $Q$.

There is also the possibility of waiting until the end of the episode before sampling from $D(s,a,r,s')$. This can allow you to ground your reward $r$ at each step, in case the reward is delayed.

  • $\begingroup$ great, and one more question, do I sample just one transition at the end of each episode, or many? Thanks! $\endgroup$ – naimelhajj Nov 29 '16 at 14:10
  • 1
    $\begingroup$ what exactly do you mean by ground ? do I add them up? $\endgroup$ – naimelhajj Nov 29 '16 at 15:48
  • $\begingroup$ With online training you use the predicted future reward. So after each step you record $r$, and you update $Q$ such that $Q(s)$ moves toward $r+Q(s')$. $\endgroup$ – KPLauritzen Nov 29 '16 at 23:33
  • $\begingroup$ If, for example, all the reward is given at the last step of the episode, it doesn't help much to do online learning. You might be better of running the episode to see what reward you get and then update $Q$ with the discounted future reward. $\endgroup$ – KPLauritzen Nov 29 '16 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.