One part from the bellman's equation in Sutton's "Reinforcement Learning: An introduction" confuses me: enter image description here

the third line contains the term $\sum_{s'}\sum_r{p(s', r|s, a)}$. The fact that it involves $\sum_r$ makes me believe that given a state $s'$, we need to sum up all of the rewards we might receive from $s'$, meaning that there might be more than one possible rewards we might receive when we reach $s'$. But isn't the reward we receive when we arrive at a state fixed?

If there is indeed more than one possible reward to be received at $s'$, then value under policy $\pi$ at $s'$ ,$v_{\pi}(s')$, is $E_{\pi}[G_{t+1}|S_{t+1} = s']$, and $G_{t+1} $ is not dependent on the reward received at $S_{t+1}=s'$ since $G_{t+1} = \sum^{\infty}_{k=0}{\gamma}^kR_{t+k+2}$. Then shouldn't the final equation be $$\sum_a{\pi(a|s)}\sum_{s'}{p(s'|s, a)\gamma v_\pi(s')\sum_rp(r|s')r}$$ where $p(r|s')$ is the probability of receiving reward $r$ at state $s'$?


But isn't the reward we receive when we arrive at a state fixed?

Not necessarily -- it can be a stochastic reward.

I'm not sure how you arrived at your final equation, but in general, the reward is dependent on $s$, $s'$, and $a$, so you can't factor $p(s', r|s,a)$ into $p(r|s') p(s'|s,a)$ like it seems you tried to. At best, you can write $p(s', r|s, a) = p(s'|s,a) p(r|s', s, a)$, which isn't immediately useful.

  • $\begingroup$ So if I factor it out like that, the final equation should become $\sum_a{\pi(a|s)}\sum_{s'}p(s'|s,a)\gamma v_\pi(s')\sum_r{p(r|s', s, a)}r$ ? Is this the same as the second last equation from the picture? $\endgroup$ – PsychoCom Aug 9 '18 at 8:08
  • $\begingroup$ @PsychoCom i think you may have made an algebra mistake somwehere -- that doesn't look equivalent -- it looks like you dropped an addition sign somewhere? $\endgroup$ – shimao Aug 9 '18 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.