I was trying to understand MDP's in the context of reinforcement learning, specifically I was trying to understand what the reward function explicitly depends on.

I have seen a formulation of the reward function as defined by Andrew Ng in his lecture notes as:

$$R: S \times A \mapsto \mathbb{R}$$

Which means that the the reward function depends on the current state and the action take at that state and maps to some real number (the reward).

To get a different perspective, I read the interpretation wikipedia had:

The process responds at the next time step by randomly moving into a new state s', and giving the decision maker a corresponding reward $R_a(s,s')$. Which seems to be a different interpretation in my opinion since this would make the reward function more of a function of the form:

$$R: S \times A \times S\mapsto \mathbb{R}$$

Which in my opinion, seems to be a completely different thing. I was trying to understand if the two formulations were actually the same (and if it was possible to prove their equivalence) in the context of MDP's applied to reinforcement learning.


The two definitions are not the same, but it essentially boils down to a modelling choice: for some problems, the reward function might be easier to define on the (state,action) pairs, while for others, the tuple (state,action,state) might be more appropriate. There's even a third option that only defines the reward on the current state (this can also be found in some references).

I do think the definition of the reward function R(s,a) on the (state, action) pair is the most common, however. But the core learning algorithms remain the same whatever your exact design choice for the reward function.

  • $\begingroup$ Basically, it comes down to a modeling choice right? Depending on what actually is feasible to learn? If say R(s,a) is too slow and R(s) is much faster, even if its less accurate, we might choose R(s)? Right? Its basically a choice depending on the application? $\endgroup$ – Pinocchio Jan 10 '15 at 21:19
  • $\begingroup$ It's not really that one representation is "faster" or "more accurate" than another. It's more than the type of function depends on the domain you are trying to model. For instance, if you simply want to encode in your reward function that some states are "good" (goal states) while others are "bad" (failure states), then the easiest is to encode the reward function as R(s). But if the reward also depend on the system action, then the function R(s,a) is to be preferred. Finally, if the resulting state s' should also be accounted for when determining the reward, R(s,a,s') is best. $\endgroup$ – Pierre Lison Jan 11 '15 at 16:05

In addition to Pierre Lison's answer in favor of a reward function as $ R: S \times A \rightarrow \mathbb{R} $, Sutton and Barto touch on the topic in chapter 3.6 of their book "Reinforcement Learning: An Introduction".

Although the accepted answer is correct in terms of what is most commonly used, they prefer $ \mathcal{R}: S \times A \times S \rightarrow \mathbb{R} $. From said chapter:

In conventional MDP theory, $\mathcal{R}_{ss'}^a $ always appears in an expected value sum [...], and therefore it is easier to use $R_s^a$. In reinforcement learning, however, we more often have to refer to individual actual or sample outcomes. In teaching reinforcement learning, we have found the notation $\mathcal{R}_{ss'}^a $ to be more straightforward conceptually and easier to understand.

  • $\begingroup$ $R(s,a,s')$ is better according to them because its easier to comprehend? $\endgroup$ – Pinocchio Mar 12 '15 at 18:27
  • $\begingroup$ with $S \times A \times S $ you mention each component explicitly, while with $S \times A $ you implicitly denote the sum of successive rewards, weighted by probability. as they say: $\mathcal{R}^a_s = \sum_{s'}^S \mathcal{P}_{ss'}^a \mathcal{R}_{ss'}^a $. btw: they probably mean the same by $R $ and $\mathcal{R} $. edit: latex $\endgroup$ – wehnsdaefflae Mar 12 '15 at 18:41

I think $R(s,a,s')$ is the same thing as $R(s,a)$ in the MDP setting because $s'$ is determined by the transition function $T(s,a)$. Therefore $R(s,a,s')$ becomes $R(s,a,T(s,a))$, which can be simplified as $R(s,a)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.