4
$\begingroup$

I'm trying to understand exactly how to use experience replay in DQN but I'm not sure I understand how it's done.

Here's how I think it develops:

Observe state s
Take action a
Observe reward r and new state s'
Store transition in replay memory D(s,a,r,s')

Now my question is where exactly do we go from here? do we train the network and update the weights here, then on the next transition, instead of training the network on that transition, we sample a random one from the replay memory and update the weights like below?

Update weights
take action a
observe reward r and new state s'
Store new transition in D
sample random transition from D (which till now has only 2 entries)
Update weights according to sampled transition
repeat...

Is this how it's done?

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you spell out DQN? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Nov 29, 2016 at 14:07
  • $\begingroup$ @kjetilbhalvorsen If I do, will you help? :D $\endgroup$
    – naimelhajj
    Commented Nov 29, 2016 at 14:12

1 Answer 1

Reset to default
3
$\begingroup$

What you are describing at the end is online learning, where we are continually updating the approximation of $Q$.

There is also the possibility of waiting until the end of the episode before sampling from $D(s,a,r,s')$. This can allow you to ground your reward $r$ at each step, in case the reward is delayed.

Improve this answer
$\endgroup$
4
  • $\begingroup$ great, and one more question, do I sample just one transition at the end of each episode, or many? Thanks! $\endgroup$
    – naimelhajj
    Commented Nov 29, 2016 at 14:10
  • 1
    $\begingroup$ what exactly do you mean by ground ? do I add them up? $\endgroup$
    – naimelhajj
    Commented Nov 29, 2016 at 15:48
  • $\begingroup$ With online training you use the predicted future reward. So after each step you record $r$, and you update $Q$ such that $Q(s)$ moves toward $r+Q(s')$. $\endgroup$
    – KPLauritzen
    Commented Nov 29, 2016 at 23:33
  • $\begingroup$ If, for example, all the reward is given at the last step of the episode, it doesn't help much to do online learning. You might be better of running the episode to see what reward you get and then update $Q$ with the discounted future reward. $\endgroup$
    – KPLauritzen
    Commented Nov 29, 2016 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.