# When to choose SARSA vs. Q Learning

SARSA and Q Learning are both reinforcement learning algorithms that work in a similar way. The most striking difference is that SARSA is on policy while Q Learning is off policy. The update rules are as follows:

Q Learning:

$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$

SARSA:

$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+1},a_{t+1})−Q(s_t,a_t)]$$

where $$s_t,\,a_t$$ and $$r_t$$ are state, action and reward at time step $$t$$ and $$\gamma$$ is a discount factor.

They mostly look the same except that in SARSA we take actual action and in Q Learning we take the action with highest reward.

Are there any theoretical or practical settings in which one should the prefer one over the other? I can see that taking the maximum in Q Learning can be costly and even more so in continuous action spaces. But is there anything else?

• In continuous action spaces, direct policy search methods such as various policy-gradient methods are commonly used since—as you have figured out—maintaning and evaluating a discrete value function for a continuous action space is unpractical, especially when the action space has many dimensions (because of the curse of dimensionality). – HelloGoodbye Sep 20 '18 at 23:13

They mostly look the same except that in SARSA we take actual action and in Q Learning we take the action with highest reward.

Actually in both you "take" the actual single generated action $a_{t+1}$ next. In Q learning, you update the estimate from the maximum estimate of possible next actions, regardless of which action you took. Whilst in SARSA, you update estimates based on and take the same action.

This is probably what you meant by "take" in the question, but in the literature, taking an action means that it becomes the value of e.g. $a_{t}$, and influences $r_{t+1}$, $s_{t+1}$.

Are there any theoretical or practical settings in which one should the prefer one over the other?

• Q-learning directly learns the optimal policy, whilst SARSA learns a near-optimal policy whilst exploring. If you want to learn an optimal policy using SARSA, then you will need to decide on a strategy to decay $\epsilon$ in $\epsilon$-greedy action choice, which may become a fiddly hyperparameter to tune.