Skip to main content
Cross Validated

Return to Question

added 456 characters in body
Source Link
Alberto
Alberto
  • 1.4k
  • 3
  • 12

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why not should SARSA also be an off-policy algorithm?

Edit: since seems like it's that clear what is the MC estimate I'm referring to, for some reason, my question is:

the two update differ only by an expected value, and $Q(s', a'\sim\pi(s')) \approx \mathbb{E}_{a\sim\pi(s')}[Q(s', a)]$, since it is a one sample monte carlo estimate, which is an unbiased estimate, so there is no reason why one should be on policy and the other off policy, since in expectation they lead to the same update

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why not should SARSA also be an off-policy algorithm?

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why not should SARSA also be an off-policy algorithm?

Edit: since seems like it's that clear what is the MC estimate I'm referring to, for some reason, my question is:

the two update differ only by an expected value, and $Q(s', a'\sim\pi(s')) \approx \mathbb{E}_{a\sim\pi(s')}[Q(s', a)]$, since it is a one sample monte carlo estimate, which is an unbiased estimate, so there is no reason why one should be on policy and the other off policy, since in expectation they lead to the same update

edited body
Source Link
Alberto
Alberto
  • 1.4k
  • 3
  • 12

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why not should SARSA also not be an off-policy algorithm?

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why should SARSA also not be an off-policy algorithm?

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why not should SARSA also be an off-policy algorithm?

Source Link
Alberto
Alberto
  • 1.4k
  • 3
  • 12

Is Expected Sarsa is off-policy, and SARSA is just an MC estimate of Expected SARSA, why is it on-policy?

So, expected SARSA defines the update as: $$ Q(s,a) = Q(s,a) +\alpha (R+ \mathbb{E}_{a\sim\pi(s')}[Q(s', a)] - Q(s,a)) $$ Where SARSA defines the update as $a'\sim\pi(s')$: $$ Q(s,a) = Q(s,a) +\alpha (R+ Q(s', a') - Q(s,a)) $$

So how is SARSA not just a MC estimate of ExpSARSA? and since MC is unbiased, why should SARSA also not be an off-policy algorithm?