2
$\begingroup$

I am reading Sutton Barto on Reinforcement Learning. I understand that $TD(\lambda)$ methods propose better performance than Monte Carlo methods, with TD methods combining advantages of Dynamic Programming and Monte Carlo approaches.

The transition to Q-methods : SARSA, Q-Learning, Expected SARSA is unclear to me. Which gaps is filled by Q-methods, what is the rational behind their use rather than TD methods ?

$\endgroup$
1
$\begingroup$

The transition to Q-methods : SARSA, Q-Learning, Expected SARSA is unclear to me. Which gaps is filled by Q-methods, what is the rational behind their use rather than TD methods ?

There isn't really such a thing as "Q methods" in the sense that you are using it. All of SARSA, Q-learning and Expected SARSA are TD methods. Specifically they are using TD approach in the context of action values, which is necessary for value based model-free control problems, because if you just used state values $V(s)$ then you have no way to generate your best guess at the optimal policy. Using $Q(s,a)$ immediately gives you $\pi(s) = \text{max}_a Q(s,a)$.

Model-free Monte Carlo control likewise must use estimates of $Q(s,a)$ which is why there is not really a clear class of methods called "Q methods", although it is a matter of convention what all these things are called and they are all inter-related in any case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.