Episodic Semi-gradient Q-learning for Estimating approximation of optimal action-value function

Question

at page 244 of Sutton and Barto book on Reinforcement Learning (book) is described the pseudocode for episodic semi-gradient Sarsa, while it is never given a pseudocode for the corresponding episodic semi-gradient Q-learning. I am aware of the issues related to the deadly triad (function approximation, bootstrapping, off-policy training), but still I am interested in understanding, for theoretical reasons, how a pseudocode for episodic semi-gradient Q-learning should be written, so I am asking if the following one is correct:

Input: a differentiable action-value function parameterization $\hat{q}(s,a,\mathbf{w})$

Algorithm parameters: step size $\alpha >0$, small $\varepsilon >0$.

Initialize $\mathbf{w} \in \mathbb{R}^d$ arbitrarily

Loop for each episode:

$\hskip 1cm S \leftarrow $ initial state of episode

$\hskip 1cm$ Loop for each step of episode

$\hskip 2cm$ Choose $A$ as a function of $\hat{q}(S,.,\mathbf{w})$ (e.g $\varepsilon$-greedy)

$\hskip 2cm$ Take action $A$, observe $R,S'$

$\hskip 2cm$ If $S'$ is terminal

$\hskip 3cm \mathbf{w} \leftarrow \mathbf{w} + \alpha \left[R-\hat{q}(S,A,\mathbf{w})\right] \nabla \hat{q}(S,A,\mathbf{w})$

$\hskip 3cm$ Go to next episode

$\hskip 2cm$ Else

$\hskip 3cm \mathbf{w} \leftarrow \mathbf{w} + \alpha \left[R + \gamma \max_a \hat{q}(S',a,\mathbf{w}) -\hat{q}(S,A,\mathbf{w})\right] \nabla \hat{q}(S,A,\mathbf{w})$

$\hskip 2cm S \leftarrow S'$

In particular I am interested on the different way to update weights between Sarsa and Q-learning: if I am not wrong, Sarsa update weights only after $A'$ has been determined using the current estimates of $\mathbf{w}$, while, if my pseudocode is correct, Q-learning update weights before $A'$ has been determined, and so $A'$ is determined using the new estimate of $\mathbf{w}$.

Are there any errors in what I have written?

Answer 1

I see that the question is pretty old, but since there is no answer, let me give you my opinion.

First of all, as you said Q-Learning is off-policy and TD hence deadly triad if used with function approximation.

If you really wanted to write it anyway; I am confused about your explanation on the bottom part. I believe off-policy has not much difference than SARSA, except that now the agent won't be acting towards the policy we are using, but acting according to the trajectory on behavior policy. So the difference is not about updating weights before A' or not. If I understood it correctly.