# Episodic Semi-gradient Q-learning for Estimating approximation of optimal action-value function

at page 244 of Sutton and Barto book on Reinforcement Learning (book) is described the pseudocode for episodic semi-gradient Sarsa, while it is never given a pseudocode for the corresponding episodic semi-gradient Q-learning. I am aware of the issues related to the deadly triad (function approximation, bootstrapping, off-policy training), but still I am interested in understanding, for theoretical reasons, how a pseudocode for episodic semi-gradient Q-learning should be written, so I am asking if the following one is correct:

Input: a differentiable action-value function parameterization $$\hat{q}(s,a,\mathbf{w})$$

Algorithm parameters: step size $$\alpha >0$$, small $$\varepsilon >0$$.

Initialize $$\mathbf{w} \in \mathbb{R}^d$$ arbitrarily

Loop for each episode:

$$\hskip 1cm S \leftarrow$$ initial state of episode

$$\hskip 1cm$$ Loop for each step of episode

$$\hskip 2cm$$ Choose $$A$$ as a function of $$\hat{q}(S,.,\mathbf{w})$$ (e.g $$\varepsilon$$-greedy)

$$\hskip 2cm$$ Take action $$A$$, observe $$R,S'$$

$$\hskip 2cm$$ If $$S'$$ is terminal

$$\hskip 3cm \mathbf{w} \leftarrow \mathbf{w} + \alpha \left[R-\hat{q}(S,A,\mathbf{w})\right] \nabla \hat{q}(S,A,\mathbf{w})$$

$$\hskip 3cm$$ Go to next episode

$$\hskip 2cm$$ Else

$$\hskip 3cm \mathbf{w} \leftarrow \mathbf{w} + \alpha \left[R + \gamma \max_a \hat{q}(S',a,\mathbf{w}) -\hat{q}(S,A,\mathbf{w})\right] \nabla \hat{q}(S,A,\mathbf{w})$$

$$\hskip 2cm S \leftarrow S'$$

In particular I am interested on the different way to update weights between Sarsa and Q-learning: if I am not wrong, Sarsa update weights only after $$A'$$ has been determined using the current estimates of $$\mathbf{w}$$, while, if my pseudocode is correct, Q-learning update weights before $$A'$$ has been determined, and so $$A'$$ is determined using the new estimate of $$\mathbf{w}$$.

Are there any errors in what I have written?