1
$\begingroup$

I'm working on the RL book of Barto and Sutton, the author has provided the proof based on the policy improvement theorem, I can fully understand the inequality, but for the first equality, it really confuses me. why does $ q_{\pi}(s,\pi^{'}(s)) = \sum_{a}\pi^{'}(a|s)q(s,a)$ holds? I guess the reason here is because $\pi^{'}$ is stochastic, but where the summation come from?

enter image description here

$\endgroup$
1
  • $\begingroup$ @Neil Slater I have posted the question here, thank you for the help $\endgroup$ Oct 20, 2020 at 1:21

1 Answer 1

1
$\begingroup$

On page 96, the formula is stated.

It defines how can we evaluate the $q$-value for the policy $\pi'(s)$.

Given state $s$, $\pi'(a|s)$ is the probability that action $a$ is taken given that we are at state $s$ and if we take action $a$, we should expect a $q$-value of $q_{\pi}(s,a)$. We should then consider all possible cases and sum them up due to the total law of expectation.

$$E(R|S)=\sum_a P(A=a|S)E(R|s,A=a)$$

where $R$ is the reward, $S$ is the state, and $A$ is the action.

$\endgroup$
2
  • $\begingroup$ Appreciate the help, yes, I understand now. But based on the definition, $q_{\pi}(s ,a)=E(R|S,A)$, so $q_{\pi}(s ,{\pi}')$ should be something like $E(R|S,A'={\pi}')$? $\endgroup$ Oct 20, 2020 at 3:17
  • $\begingroup$ I think on page $96$, the meaning of evaluating the $q$ value when the action is replaced by a policy is defined and it has to be defined based on entities that have been defined earlier. $q(s,a)$ is defined earlier. $\endgroup$ Oct 20, 2020 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.