0
$\begingroup$

In section 4.2 of Sutton and Barto's book, they say that to decide whether a policy $\pi$ needs to be changed, we can compare the action value function $q_\pi(s, \pi'(s))$ and $v_\pi(s)$. Why can't we instead compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$? This would not make a difference since the policies are deterministic, right?

$\endgroup$
3
  • $\begingroup$ My understanding is that one has the relationship $$ v_\pi(s) = \mathbb{E}_\pi[R_t\mid s_t=s] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ So in a sense $v_\pi(s)$ weights all possible decisions and their expected reward using the probability of doing $a$ in state $s$ according to policy $\pi$, which is $\pi(s,a)$. How do you define $\pi(s)$ and $\pi'(s)$ ? Is this the greedy assignment $\pi(s) = \arg\max_a\pi(s,a)$ ? $\endgroup$ Feb 23, 2021 at 13:55
  • $\begingroup$ The book mentions that the policies are deterministic. This would mean that there is only one action taken in a state with probability 1. Furthermore, it is mentioned that "For some state s we would like to know whether or not we should change the policy to deterministically choose an action $a \neq \pi(s)$." So, the two policies differ in that in state s, policy $\pi'$ takes an action $a \neq \pi(s)$. And yes, $\pi(s) = \arg\max_a\pi(s,a)$. $\endgroup$ Feb 23, 2021 at 14:07
  • $\begingroup$ If your policy is deterministic then $v_\pi(s) = q_\pi(s,\pi(s))$ ! $\endgroup$ Feb 23, 2021 at 14:09

1 Answer 1

0
$\begingroup$

If your policy is deterministic then $v_\pi(s) = q_\pi(s,\pi(s))$. This is because the expectation in $$ v_\pi(s) = \mathbb{E}_{a\sim\pi}\left[q_\pi(s,a)\mid s_t=s\right] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ is taken over a deterministic policy = $\pi(s,a) = \delta_{\pi(s)}(a)$ and therefore the expectation becomes an evaluation at $a=\pi(s)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.