I am referring to pages 130-131 of Sutton and Barto book on Reinforcement Learning available here: book

I don't understand the slight difference that there is between the two procedural algorithms described respectively at page 130 for Sarsa and at page 131 for Q-learning.

Indeed, in the first case the $\varepsilon$-greedy choice of action $A$ is inside the loop for each episode but before of the loop for each step of the episode, while in the second one the $\varepsilon$-greedy choice of action $A$ is inside the loop for each step of the episode. Does this imply any real difference between the two algorithms (except the update rule for $Q(s,a)$ of course), or is this only a formal one?

To be more precise: can I move the the $\varepsilon$-greedy choice of action $A$ inside the loop for each step of the episode also in Sarsa algorithm?


1 Answer 1


No, we cannot. Otherwise, the already-determined next action $A'$ would be thrown away.

In SARSA, next action $A'$ is selected in the middle of current step loop, and it replaces $A$ in the next step (more precisely, at the end of current step). In other words, at the beginning of next step, $A$ should be the already-selected action $A'$ from current step, we cannot throw $A'$ away by selecting $A$ from a new ε-greedy search. On the other hand, in Q-learning, only one action is involved in each loop, thus, if the only action $A$ is selected at the beginning of each step, nothing is thrown away.

  • $\begingroup$ Thanks for your answer. How would you rewrite the pseudocode for SARSA algorithm in the case of a continuing task, where there is no initial state and action of the episode? $\endgroup$
    – hardhu
    Mar 19, 2019 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.