Filling replay memory randomly at the beginning or not? I have a question about the correct way of implementing replay memory for a DQN. Assuming the the maximum size of memory is $N$, what do we need to do if the batch size is $K$?
1) Fill the memory just by choosing $N$ random actions, without updating the neural network. Then, take batch of samples with size $K$,...
2) When we add each experience to the memory, update the neural network with a batch of size $\min \{K, \text{current size of memory} \}$ and do not wait for the memory to get filled completely.
Does this really matter?
Thanks,
 A: Your option (2) is not widely used, because the initial mini-batches would be too correlated, and more likely to make initial stability problems in Q-learning worse than fix them. So it is wise to wait a bit longer until a randomly sampled mini-batch would be less biased. In addition, for a lot of environments there could be very little to gain until the agent has seen at least one episode end - for example any environment with an end result of success or failure has no real "signal" to learn from until it has seen the end reward different from zero.
There is an option (3) as a compromise:


*Add examples to memory without sampling initially. After $M \lt N$ actions, start taking minibatches of size $K \lt M$ 


And also an option (4) which works for some episodic problems:


*Add examples to memory without sampling initially. After $M$ episodes, start taking minibatches of size $K \lt M \times EpisodeLength$ 


This is like many hyper-parameter choices, a case of try-it-and-see. Even your option (2) will probably work well enough in the long term, the difference will be seen in learning curves, and may interact with other hyper-parameters, such as learning rate.
