# Comparing action values for policy improvement (Sutton and Barto)

In section 4.2 of Sutton and Barto's book, they say that to decide whether a policy $$\pi$$ needs to be changed, we can compare the action value function $$q_\pi(s, \pi'(s))$$ and $$v_\pi(s)$$. Why can't we instead compare $$q_\pi(s, \pi'(s))$$ and $$q_\pi(s, \pi(s))$$? This would not make a difference since the policies are deterministic, right?

New contributor
01110000_01110000 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
• My understanding is that one has the relationship $$v_\pi(s) = \mathbb{E}_\pi[R_t\mid s_t=s] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ So in a sense $v_\pi(s)$ weights all possible decisions and their expected reward using the probability of doing $a$ in state $s$ according to policy $\pi$, which is $\pi(s,a)$. How do you define $\pi(s)$ and $\pi'(s)$ ? Is this the greedy assignment $\pi(s) = \arg\max_a\pi(s,a)$ ? – ArnoV Feb 23 at 13:55
• The book mentions that the policies are deterministic. This would mean that there is only one action taken in a state with probability 1. Furthermore, it is mentioned that "For some state s we would like to know whether or not we should change the policy to deterministically choose an action $a \neq \pi(s)$." So, the two policies differ in that in state s, policy $\pi'$ takes an action $a \neq \pi(s)$. And yes, $\pi(s) = \arg\max_a\pi(s,a)$. – 01110000_01110000 Feb 23 at 14:07
• If your policy is deterministic then $v_\pi(s) = q_\pi(s,\pi(s))$ ! – ArnoV Feb 23 at 14:09

## 1 Answer

If your policy is deterministic then $$v_\pi(s) = q_\pi(s,\pi(s))$$. This is because the expectation in $$v_\pi(s) = \mathbb{E}_{a\sim\pi}\left[q_\pi(s,a)\mid s_t=s\right] = \int_\mathcal{A}q_\pi(s,a)\pi(s,a)\,\mathrm{d}a$$ is taken over a deterministic policy = $$\pi(s,a) = \delta_{\pi(s)}(a)$$ and therefore the expectation becomes an evaluation at $$a=\pi(s)$$.