In section 4.2 of Sutton and Barto's book, they say that to decide whether a policy $\pi$ needs to be changed, we can compare the action value function $q_\pi(s, \pi'(s))$ and $v_\pi(s)$. Why can't we instead compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$? This would not make a difference since the policies are deterministic, right?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ My understanding is that one has the relationship $$ v_\pi(s) = \mathbb{E}_\pi[R_t\mid s_t=s] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ So in a sense $v_\pi(s)$ weights all possible decisions and their expected reward using the probability of doing $a$ in state $s$ according to policy $\pi$, which is $\pi(s,a)$. How do you define $\pi(s)$ and $\pi'(s)$ ? Is this the greedy assignment $\pi(s) = \arg\max_a\pi(s,a)$ ? $\endgroup$– NoVariationFeb 23, 2021 at 13:55
-
$\begingroup$ The book mentions that the policies are deterministic. This would mean that there is only one action taken in a state with probability 1. Furthermore, it is mentioned that "For some state s we would like to know whether or not we should change the policy to deterministically choose an action $a \neq \pi(s)$." So, the two policies differ in that in state s, policy $\pi'$ takes an action $a \neq \pi(s)$. And yes, $\pi(s) = \arg\max_a\pi(s,a)$. $\endgroup$– 01110000_01110000Feb 23, 2021 at 14:07
-
$\begingroup$ If your policy is deterministic then $v_\pi(s) = q_\pi(s,\pi(s))$ ! $\endgroup$– NoVariationFeb 23, 2021 at 14:09
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If your policy is deterministic then $v_\pi(s) = q_\pi(s,\pi(s))$. This is because the expectation in $$ v_\pi(s) = \mathbb{E}_{a\sim\pi}\left[q_\pi(s,a)\mid s_t=s\right] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ is taken over a deterministic policy = $\pi(s,a) = \delta_{\pi(s)}(a)$ and therefore the expectation becomes an evaluation at $a=\pi(s)$.