I have a question about the correct way of implementing replay memory for a DQN. In the pseudo code provided in the original DQN network, it seems they update the network at each step (after adding a new experience to the replay memory and taking a batch of samples from it). But, as claimed by this paper here, they update the network every 4 steps:
Another parameter worth noting is the network update frequency. The original DQN implementation only chose to take a gradient descent step every 4 environment steps of the algorithm as opposed to every step, as Algorithm 1 might suggest. Not only does this greatly increase the training speed (since learning steps on the network are far more expensive than forward passes), it also causes the experience memory to more closely resemble the state distribution of the current policy (since 4 new frames are added to the memory between training steps as opposed to 1) and may prevent the network from over-fitting.
Is this really correct? If yes, why 4? Assume the environment we are going to learn is such that at least every $K$ steps, we might observe a noticeable positive reward. Can we change 4 to $K$?