I have it from relevant authority that MDP formulation of reinforcement learning is:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $r_t \sim R(.|s_t, a_t)$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $r_t$ and next state $s_{t+1}$

Question: Is the reward not a single number given $a_t, s_{t+1}$ and $s_t$? If we are sampling $s_{t+1}$ then sampling reward as well does not make sense to me. I think the process should instead be:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $\require{enclose} \enclose{horizontalstrike}{r_t \sim R(.|s_t, a_t)}$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $r_t|(s_t, s_{t+1}, a_t)$ and next state $s_{t+1}$

I have it from relevant authority that MDP formulation of reinforcement learning is:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $r_t \sim R(.|s_t, a_t)$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $r_t$ and next state $s_{t+1}$

Question: Is the reward not a single number given $a_t, s_{t+1}$ and $s_t$? If we are sampling $s_{t+1}$ then sampling reward as well does not make sense to me. I think the process should instead be:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $\require{enclose} \enclose{horizontalstrike}{r_t \sim R(.|s_t, a_t)}$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $\require{enclose}\enclose{horizontalstrike}r_t$ $r_t|(s_t, s_{t+1}, a_t)$ and next state $s_{t+1}$

I have it from relevant authority that MDP formulation of reinforcement learning is:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $r_t \sim R(.|s_t, a_t)$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $r_t$ and next state $s_{t+1}$

Question: Is the reward not a single number given $s_{t+1}$ and $s_t$? If we are sampling $s_{t+1}$ then sampling reward as well does not make sense to me. I think the process should instead be:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $\require{enclose} \enclose{horizontalstrike}{r_t \sim R(.|s_t, a_t)}$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $r_t|(s_t, s_{t+1})$ and next state $s_{t+1}$

I have it from relevant authority that MDP formulation of reinforcement learning is:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $r_t \sim R(.|s_t, a_t)$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $r_t$ and next state $s_{t+1}$

Question: Is the reward not a single number given $a_t, s_{t+1}$ and $s_t$? If we are sampling $s_{t+1}$ then sampling reward as well does not make sense to me. I think the process should instead be:

1. At time step t = 0 environemnt samples initial state $s_0 \sim p(s_0)$
2. Then, for t = 0 until done:

• Agent selects action $a_t$
• Environment samples reward $\require{enclose} \enclose{horizontalstrike}{r_t \sim R(.|s_t, a_t)}$
• Environment samples next state $s_{t+1} \sim P(.|s_t, a_t)$
• Agent receives reward $r_t|(s_t, s_{t+1}, a_t)$ and next state $s_{t+1}$