0
$\begingroup$

In section 4.2 of Sutton and Barto's book, they say that to decide whether a policy $\pi$ needs to be changed, we can compare the action value function $q_\pi(s, \pi'(s))$ and $v_\pi(s)$. Why can't we instead compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$? This would not make a difference since the policies are deterministic, right?

New contributor
01110000_01110000 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
3
  • $\begingroup$ My understanding is that one has the relationship $$ v_\pi(s) = \mathbb{E}_\pi[R_t\mid s_t=s] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ So in a sense $v_\pi(s)$ weights all possible decisions and their expected reward using the probability of doing $a$ in state $s$ according to policy $\pi$, which is $\pi(s,a)$. How do you define $\pi(s)$ and $\pi'(s)$ ? Is this the greedy assignment $\pi(s) = \arg\max_a\pi(s,a)$ ? $\endgroup$ – ArnoV Feb 23 at 13:55
  • $\begingroup$ The book mentions that the policies are deterministic. This would mean that there is only one action taken in a state with probability 1. Furthermore, it is mentioned that "For some state s we would like to know whether or not we should change the policy to deterministically choose an action $a \neq \pi(s)$." So, the two policies differ in that in state s, policy $\pi'$ takes an action $a \neq \pi(s)$. And yes, $\pi(s) = \arg\max_a\pi(s,a)$. $\endgroup$ – 01110000_01110000 Feb 23 at 14:07
  • $\begingroup$ If your policy is deterministic then $v_\pi(s) = q_\pi(s,\pi(s))$ ! $\endgroup$ – ArnoV Feb 23 at 14:09
0
$\begingroup$

If your policy is deterministic then $v_\pi(s) = q_\pi(s,\pi(s))$. This is because the expectation in $$ v_\pi(s) = \mathbb{E}_{a\sim\pi}\left[q_\pi(s,a)\mid s_t=s\right] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ is taken over a deterministic policy = $\pi(s,a) = \delta_{\pi(s)}(a)$ and therefore the expectation becomes an evaluation at $a=\pi(s)$.

$\endgroup$

Your Answer

01110000_01110000 is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.