I have a question about the correct way of implementing replay memory for a DQN. Assuming the the maximum size of memory is $N$, what do we need to do if the batch size is $K$?

1) Fill the memory just by choosing $N$ random actions, without updating the neural network. Then, take batch of samples with size $K$,...

2) When we add each experience to the memory, update the neural network with a batch of size $\min \{K, \text{current size of memory} \}$ and do not wait for the memory to get filled completely.

Does this really matter?



Your option (2) is not widely used, because the initial mini-batches would be too correlated, and more likely to make initial stability problems in Q-learning worse than fix them. So it is wise to wait a bit longer until a randomly sampled mini-batch would be less biased. In addition, for a lot of environments there could be very little to gain until the agent has seen at least one episode end - for example any environment with an end result of success or failure has no real "signal" to learn from until it has seen the end reward different from zero.

There is an option (3) as a compromise:

  1. Add examples to memory without sampling initially. After $M \lt N$ actions, start taking minibatches of size $K \lt M$

And also an option (4) which works for some episodic problems:

  1. Add examples to memory without sampling initially. After $M$ episodes, start taking minibatches of size $K \lt M \times EpisodeLength$

This is like many hyper-parameter choices, a case of try-it-and-see. Even your option (2) will probably work well enough in the long term, the difference will be seen in learning curves, and may interact with other hyper-parameters, such as learning rate.

  • $\begingroup$ But I guess the original DQN paper suggest option (1), right? $\endgroup$ – user491626 Jun 20 '18 at 20:59
  • $\begingroup$ @user491626: I'm not sure, I would need to check the paper. If the pool size is relatively low (e.g. a few thousand records), then there is not much difference in practice between (1) and (3) here $\endgroup$ – Neil Slater Jun 20 '18 at 21:02
  • $\begingroup$ @user491626: Checking the paper here: cs.toronto.edu/~vmnih/docs/dqn.pdf - there is no mention of any delay parameter or minimum replay size, so it looks like they used your option (2). It might be an omitted detail though . . . $\endgroup$ – Neil Slater Jun 20 '18 at 21:07
  • $\begingroup$ Thanks for your response. Yes, It seems they didn't mention all the details. This paper arxiv.org/abs/1711.07478 says they used option (1). One more, question, this paper says instead of updating the network every 1 step, they do that every 4 steps. In term of speed, this would make the training faster and this paper says, it also better stabilizes the network. Is this right? $\endgroup$ – user491626 Jun 20 '18 at 21:45
  • $\begingroup$ it was a separate question. I asked it as another question... $\endgroup$ – user491626 Jun 20 '18 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.