I'm working on the RL book of Barto and Sutton, the author has provided the proof based on the policy improvement theorem, I can fully understand the inequality, but for the first equality, it really confuses me. why does $ q_{\pi}(s,\pi^{'}(s)) = \sum_{a}\pi^{'}(a|s)q(s,a)$ holds？ I guess the reason here is because $\pi^{'}$ is stochastic, but where the summation come from?
On page 96, the formula is stated.
It defines how can we evaluate the $q$-value for the policy $\pi'(s)$.
Given state $s$, $\pi'(a|s)$ is the probability that action $a$ is taken given that we are at state $s$ and if we take action $a$, we should expect a $q$-value of $q_{\pi}(s,a)$. We should then consider all possible cases and sum them up due to the total law of expectation.
$$E(R|S)=\sum_a P(A=a|S)E(R|s,A=a)$$
where $R$ is the reward, $S$ is the state, and $A$ is the action.
