3
$\begingroup$

I read a lot about replay memory, but did not find the material, where this method is described in detail. Tell me, please, do I understand the algorithm correctly? And if not, tell me, please, where I'm wrong.

  1. Store the experience in the replay memory (size N), until it is full.
  2. When memory is full, fill the buffer (size B) with random samples from memory.
  3. Select mini-batches (size M) from the buffer, accumulate the gradient and update weights (when the accumulated gradient = M), until the buffer is empty.

That is, to update the weights, we use not all the memory, but only a part (do random sampling from the memory for training after each episode). And when replay memory is full, overwrite the old values.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

That is, to update the weights, we use not all the memory, but only a part (do random sampling from the memory for training after each episode). And when replay memory is full, overwrite the old values.

That's correct, that's the most common strategy.

I find this code to be quite educational to understand experience replay: https://github.com/yanpanlau/Keras-FlappyBird/blob/master/qlearn.py

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you,can you also tell me please, am I right about accumulating gradient by the size of the minibatch? This step we must perform when do minibatch training. But when I accumulate gradient, my network approximates function very poorly, but when updating weights without accumulation, everything looks better (but its will be online training). $\endgroup$
    – Adam
    May 1, 2017 at 1:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.