How does DQN parameter updates work in simulation?

Question

I've already read almost every Questions-answers and material related to DQN, deep reinforcement learning, but I'm struggling to start working on simulation.

First of all, I'm trying to code using Matlab, although the most commonly used tool is Tensorflow. So far, I've understood the procedure of DQN about how experience buffer and stochastic gradient descent works for function approximator using neural networks.

What I'm confusing is that when parameters(θ) are updated, is only one step moved to the descent direction(▽loss) of performance function (in here, the loss = (target Q - output Q)^2) in an iteration? in other words, new θ= θ- γ*▽loss, this updates only once.

or in a single iteration, parameters(θ) are iteratively updated until satisfying certain termination criterion such as ||▽loss||≤η (for some small η>0), and then the next epoch is started as selecting an action given the next state according to ε-greedy rule.

Does my question make sense? or If I'm missing some important aspect here, please improve my questions.

Answer 1

In deep learning, one iteration usually means update the parameters with the gradients for one step.

Let me give you an example to explain the training process. Suppose we have a dataset with 10,000 dataset, and we choose a mini batch size of 100. Thus, for each epoch, we have 10000/100=100 iterations. To train a model, you can set a fixed number of epochs or training until the model satisfies a convergence criterion. Say you want to train for a total of 10 epochs, that means you are going to the update the model for a total of 10*100 times.

With the above being said, one iteration for DQN update means update with one gradient step.

Hope this helps.