Skip to main content
2 of 3
edited body
Alberto
  • 1.4k
  • 3
  • 12

Is Expected Sarsa is off-policy, and SARSA is just an MC estimate of Expected SARSA, why is it on-policy?

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why not should SARSA also be an off-policy algorithm?

Alberto
  • 1.4k
  • 3
  • 12