# One small confusion on $\epsilon$-Greedy policy improvement based on Monte Carlo

I'm working on the RL book of Barto and Sutton, the author has provided the proof based on the policy improvement theorem, I can fully understand the inequality, but for the first equality, it really confuses me. why does $$q_{\pi}(s,\pi^{'}(s)) = \sum_{a}\pi^{'}(a|s)q(s,a)$$ holds？ I guess the reason here is because $$\pi^{'}$$ is stochastic, but where the summation come from?

• @Neil Slater I have posted the question here, thank you for the help Commented Oct 20, 2020 at 1:21

It defines how can we evaluate the $$q$$-value for the policy $$\pi'(s)$$.
Given state $$s$$, $$\pi'(a|s)$$ is the probability that action $$a$$ is taken given that we are at state $$s$$ and if we take action $$a$$, we should expect a $$q$$-value of $$q_{\pi}(s,a)$$. We should then consider all possible cases and sum them up due to the total law of expectation.
$$E(R|S)=\sum_a P(A=a|S)E(R|s,A=a)$$
where $$R$$ is the reward, $$S$$ is the state, and $$A$$ is the action.
• Appreciate the help, yes, I understand now. But based on the definition, $q_{\pi}(s ,a)=E(R|S,A)$, so $q_{\pi}(s ,{\pi}')$ should be something like $E(R|S,A'={\pi}')$? Commented Oct 20, 2020 at 3:17
• I think on page $96$, the meaning of evaluating the $q$ value when the action is replaced by a policy is defined and it has to be defined based on entities that have been defined earlier. $q(s,a)$ is defined earlier. Commented Oct 20, 2020 at 3:24