I'm reading the public draft (pdf) of the 2nd edition of Sutton&Barto's RL Book.

There are a few things I don't understand about the Off-policy n-step Sarsa method described in the book. You can find the pseudo-code at page 159 (177) of the pdf I linked above.

Here's a picture of the pseudo-code for your convenience:

enter image description here

Here are the 3 things I don't understand:

  1. $\rho$ considers the actions $A_{\tau+1},\ldots,A_{\tau+n-1}$. Shouldn't it also consider $A_{\tau+n}$?
  2. In the third to last line, $\rho$ multiplies both $G$ and $Q(S_\tau,A_\tau)$. Shouldn't it multiply just $G$?
  3. The second to last line says that $\pi$, if being learned, must be $\epsilon$-greedy wrt $Q$ (with $\epsilon>0$). I don't understand the reason of this restriction. Can't $\pi$ be greedy just like in Q Learning? The exploration is already taken care of by $\mu$.
  • $\begingroup$ Regarding point 2, I am having the same doubt, I actually asked the question before I found yours: ai.stackexchange.com/questions/11676/… $\endgroup$ – Antoine Savine Apr 5 '19 at 14:31
  • $\begingroup$ I even messaged Rich Sutton, hopefully we get an answer. $\endgroup$ – Antoine Savine Apr 5 '19 at 14:31
  • $\begingroup$ Were you able to clarify the question (since you asked yourself the same question two years ago :)? $\endgroup$ – Antoine Savine Apr 5 '19 at 14:32

1) Aτ+n is omitted since it is the state at horizon for the n-step update and estimated value for its action-value will be used so weighting by its importance ratio is skipped.

2) I was wondering the same - it even makes my action-values to grow to infinity(with rewards -1.0 for all steps and 1.0 for goal reached). My implementation for n-Step Q(σ) estimation works fine with σ=1 matching the n-Step Tree Backup as expected but for σ=0 it reproduces the same problem with non-stop growing state-action-values as in the simple Off-Policy n-Step Sarsa.

3) If it is deterministic greedy then all its action probabilities for a given state will be 0 instead of the greedy action(for which will be 1.0) and thus the importance ratio product will be non-zero only when the actions chosen by the behavior policy will match the greedy policy - I don't think this will help exploration.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.