-1
$\begingroup$

enter image description here

Ok I’m a bit confused with this code, what exactly is a time step, isn’t it like when an action is performed,it goes to the next time step, and also, the gradient descent steps is a repeat until convergence, when it eventually converges we update the parameters of the target network why????, we haven’t explored other states yet

$\endgroup$

1 Answer 1

1
$\begingroup$

MDPs are generally analyzed in discrete time, so basically time evolves in discrete steps. Think of it like the distinct ticks of a clock -- each of these corresponds to a timestep. In an MDP, an action as taken at each timestep.

In the algorithm you posted, where do you see anything about repeating until convergence? Usually in deep q-learning a single (or fixed amount) gradient step is carried out at each timestep.

$\endgroup$
18
  • $\begingroup$ Yeah my question is , is the gradient descent step calculated once or it keeps calculating till convergence for that time step? $\endgroup$
    – Chukwudi
    Commented Jun 11, 2020 at 2:05
  • $\begingroup$ Or does it only calculate once, adjust the weights for each time step $\endgroup$
    – Chukwudi
    Commented Jun 11, 2020 at 2:05
  • $\begingroup$ At each time step, there is only one weight update (or a fixer number of them) $\endgroup$
    – harwiltz
    Commented Jun 11, 2020 at 2:11
  • $\begingroup$ Ok I also have one more question,can the weights be the same across all states , like when I enter a new state and find the target value, backpropagate and change weights, does it alter the first state, the whole point is converging the Q value to the target value $\endgroup$
    – Chukwudi
    Commented Jun 11, 2020 at 2:37
  • $\begingroup$ Like what I’m trying to say is , is the weight uniform across all states , will they still converge to the same value? $\endgroup$
    – Chukwudi
    Commented Jun 11, 2020 at 2:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.