3
$\begingroup$

I would like to clarify a doubt that I have regarding SARSA. SARSA can be used for optimal control when the policy to take action $a$ is epsilon greedy. Suppose that the policy to take action $a$ is not an epsilon greedy one, but some other policy, such as a random policy. Would the Q - values learnt from SARSA be the Q values for SARSA be the Q values for the random policy ? Similar to policy evaluation.

If Q - learning were to be used in this scenario, then it is possible for Q - learning to learn an optimal policy.

$\endgroup$

1 Answer 1

2
$\begingroup$

As long as you can sample action choice from a fixed policy, then a variant of SARSA that used that policy (as opposed to anything based on the current Q values) would evaluate that policy. You can use this to evaluate a policy even if you don't know the probability function that drives the choices, SARSA just needs the ability to sample actions, and for the policy to remain the same during evaluation.

In the case of your specific question of using a random policy:

Would the Q - values learnt from SARSA be the Q values for SARSA be the Q values for the random policy ?

The answer is yes.

In the case of your question about Q learning, then the action sampling for taking the next step is separate from the update which is based on the current maximum Q value in each state. That means the answer to your question

If Q - learning were to be used in this scenario, then it is possible for Q - learning to learn an optimal policy.

Is also yes. However, there is a caveat in this case - the speed of learning may become impractially slow due to most observations not having anything useful to contribute to knowledge about the optimal state progression. Imagine for instance an agent that must escape from a very large maze with a fixed negative reward per time step - it would take a very long time for it to do so randomly, yet until it did so at least once, it would not have any observations of the terminal state that it could use to start calculating the optimal path. This is why it is common in Q learning to use epsilon greedy approach and a minimum value of epsilon such as 0.1 or 0.01 (it may start much higher, such as 1.0 for completely random, but usually is decayed down to a minimum value like this).

Also worth noting in the maze escape scenario that Q learning with the current Q table driving the behaviour policy will tend to explore away from starting states (their estimates become progressively more and more negative) - this helpful exploration doesn't occur in all environments, but can be a useful feature of Q learning in some cases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.