at page 244 of Sutton and Barto book on Reinforcement Learning (book) is described the pseudocode for episodic semi-gradient Sarsa, while it is never given a pseudocode for the corresponding episodic semi-gradient Q-learning. I am aware of the issues related to the deadly triad (function approximation, bootstrapping, off-policy training), but still I am interested in understanding, for theoretical reasons, how a pseudocode for episodic semi-gradient Q-learning should be written, so I am asking if the following one is correct:

Input: a differentiable action-value function parameterization $\hat{q}(s,a,\mathbf{w})$

Algorithm parameters: step size $\alpha >0$, small $\varepsilon >0$.

Initialize $\mathbf{w} \in \mathbb{R}^d$ arbitrarily

Loop for each episode:

$\hskip 1cm S \leftarrow $ initial state of episode

$\hskip 1cm$ Loop for each step of episode

$\hskip 2cm$ Choose $A$ as a function of $\hat{q}(S,.,\mathbf{w})$ (e.g $\varepsilon$-greedy)

$\hskip 2cm$ Take action $A$, observe $R,S'$

$\hskip 2cm$ If $S'$ is terminal

$\hskip 3cm \mathbf{w} \leftarrow \mathbf{w} + \alpha \left[R-\hat{q}(S,A,\mathbf{w})\right] \nabla \hat{q}(S,A,\mathbf{w})$

$\hskip 3cm$ Go to next episode

$\hskip 2cm$ Else

$\hskip 3cm \mathbf{w} \leftarrow \mathbf{w} + \alpha \left[R + \gamma \max_a \hat{q}(S',a,\mathbf{w}) -\hat{q}(S,A,\mathbf{w})\right] \nabla \hat{q}(S,A,\mathbf{w})$

$\hskip 2cm S \leftarrow S'$

In particular I am interested on the different way to update weights between Sarsa and Q-learning: if I am not wrong, Sarsa update weights only after $A'$ has been determined using the current estimates of $\mathbf{w}$, while, if my pseudocode is correct, Q-learning update weights before $A'$ has been determined, and so $A'$ is determined using the new estimate of $\mathbf{w}$.

Are there any errors in what I have written?


1 Answer 1


I see that the question is pretty old, but since there is no answer, let me give you my opinion.

First of all, as you said Q-Learning is off-policy and TD hence deadly triad if used with function approximation.

If you really wanted to write it anyway; I am confused about your explanation on the bottom part. I believe off-policy has not much difference than SARSA, except that now the agent won't be acting towards the policy we are using, but acting according to the trajectory on behavior policy. So the difference is not about updating weights before A' or not. If I understood it correctly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.