# How to understand the term $\sum_{s', r}p(s', r|s,a)$ in Bellman's equation

One part from the bellman's equation in Sutton's "Reinforcement Learning: An introduction" confuses me: the third line contains the term $\sum_{s'}\sum_r{p(s', r|s, a)}$. The fact that it involves $\sum_r$ makes me believe that given a state $s'$, we need to sum up all of the rewards we might receive from $s'$, meaning that there might be more than one possible rewards we might receive when we reach $s'$. But isn't the reward we receive when we arrive at a state fixed?

If there is indeed more than one possible reward to be received at $s'$, then value under policy $\pi$ at $s'$ ,$v_{\pi}(s')$, is $E_{\pi}[G_{t+1}|S_{t+1} = s']$, and $G_{t+1}$ is not dependent on the reward received at $S_{t+1}=s'$ since $G_{t+1} = \sum^{\infty}_{k=0}{\gamma}^kR_{t+k+2}$. Then shouldn't the final equation be $$\sum_a{\pi(a|s)}\sum_{s'}{p(s'|s, a)\gamma v_\pi(s')\sum_rp(r|s')r}$$ where $p(r|s')$ is the probability of receiving reward $r$ at state $s'$?

## 1 Answer

But isn't the reward we receive when we arrive at a state fixed?

Not necessarily -- it can be a stochastic reward.

I'm not sure how you arrived at your final equation, but in general, the reward is dependent on $s$, $s'$, and $a$, so you can't factor $p(s', r|s,a)$ into $p(r|s') p(s'|s,a)$ like it seems you tried to. At best, you can write $p(s', r|s, a) = p(s'|s,a) p(r|s', s, a)$, which isn't immediately useful.

• So if I factor it out like that, the final equation should become $\sum_a{\pi(a|s)}\sum_{s'}p(s'|s,a)\gamma v_\pi(s')\sum_r{p(r|s', s, a)}r$ ? Is this the same as the second last equation from the picture? – PsychoCom Aug 9 '18 at 8:08
• @PsychoCom i think you may have made an algebra mistake somwehere -- that doesn't look equivalent -- it looks like you dropped an addition sign somewhere? – shimao Aug 9 '18 at 16:34