I'm slightly confused about the proof epsilon greedy policy improvement. This is a part of the proof:

$\begin{aligned} q_{\pi}\left(s, \pi^{\prime}(s)\right) &=\sum_{a} \pi^{\prime}(a \mid s) q_{\pi}(s, a) \\ &=\frac{\varepsilon}{|\mathcal{A}(s)|} \sum_{a} q_{\pi}(s, a)+(1-\varepsilon) \max _{a} q_{\pi}(s, a) \\ & \geq \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_{a} q_{\pi}(s, a)+(1-\varepsilon) \sum_{a} \frac{\pi(a \mid s)-\frac{\varepsilon}{|\mathcal{A}(s)|}}{1-\varepsilon} q_{\pi}(s, a) \end{aligned}$

I'm stuck on the very first line itself. Why is $q_{\pi}\left(s, \pi^{\prime}(s)\right) =\sum_{a} \pi^{\prime}(a \mid s) q_{\pi}(s, a)$? I've looked at other similar questions on this site, but they dont seem to address this line of the proof.


1 Answer 1


Sutton describes it as a natural definition.

If we are at state $s$, we take action $a$ with probability $\pi'(a|s)$ according to policy $\pi'$ and we compute the expected value.

  • $\begingroup$ Yes, but I still don't understand why the action value of taking one action ($\pi'(s)$) in state $s$ is equal to the expectation of all action values in $s$. $\endgroup$ Mar 5, 2021 at 4:02
  • $\begingroup$ it is not one action right, it is action $a$ with probability $\pi'(a|s)$. $\endgroup$ Mar 5, 2021 at 5:27
  • $\begingroup$ Ohhh yes, I forgot that epsilon greedy policies may have multiple actions that can be chosen with the same probability. Thank you. $\endgroup$ Mar 5, 2021 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.