Why (and when) does one have to learn the reward function from samples in reinforcement learning?

Question

In reinforcement learning we have a reward function that informs the agent how well its current actions and states are doing. In a some what general setting the reward function is a function of three variables:

1. Current state $S$
2. Current action at current state $\pi(s) = a$
3. Next state $S'$

So it looks something like:

$$R(S, a, S')$$

What my question is (which is probably my misunderstanding), normally the person using reinforcement learning decides what is the reward. For example, it assigns 1000 points for reaching the goal, or assings -1000 points for crashing the autonomous robot. In these scenarios, its not clear to me why we would need samples to learn R. R is a priori specified and then we use our agent. Right? However, I know I am wrong because in Andrew Ng's notes he says:

Where he says that we don't know the reward function explicitly. That seems bizarre to me. I know I am wrong and I'd love if someone could clarify to me in what scenarios do we actually have to learn R from samples?

(obviously, the transition probabilities have to be learned because one does not know how the environment will make our agent move a priori).

Answer 1

In his notes, when you must "estimate them from data", he does not mean the reward function. You rarely estimate the reward function. You typically learn the value function, which estimates the immediate reward plus the temporally-discounted future reward (if the temporal discount is zero, then you are estimating the rewards). Or, you can learn Q values, which are values associated with state-action pairs.

In summary, the reward function and the true transition function is defined by the environment. The agent learns things like the transition function, Q values, and the value function.

Answer 2

I agree with the other answers that usually, the reward function is not learned directly.

The transition probabilities also don't have to be learned. The agent can learn directly the action values, or even directly the policy, with policy gradien method for instance.

There are, however techniques for which the reward and the transition probabilities have to be learned. For example, the dyna-Q algorithm (described in Sutton & Barto) maintains a model of the environment. At each time step, the agent uses the reward and state information received from the environment to update the action-values, just like in Q-learning for instance.

But it also update its own model of the environment, and then performs N other action-values updates based on that model. The supposition is that the acting->sensing loop takes some non-null time, time that we can put to good use by improving the action-values with simulated samples.

Answer 3

This is a good question which is deeper than any imprecise wording in Andrew's notes. It is true that in RL you generally do not need to learn a reward function R(S,A,S'); you need to specify it as part of the problem setup. BUT, there are algorithms (in my experience associated with the afterstate or post-decision state value function) which require the expected reward r(S,A) = E[R(S,A,S')|S,A]. Generally the texts I've seen make little comment on this and assume that r(s,a) is known just as R(S,A,S') is known. But in some cases the reward is dependent on the future state so, without a model, you need to learn this expectation. I am currently working on such a problem where the expected reward function as well as the value function need to be learned. Note that most RL algorithms do NOT require the expected reward function, but some do. See for example the discussion on pg 58 in Algorithms for Reinforcement Learning by Szepesvari.

In summary you do not need to learn the reward function, but when working with post-decision state variables, you may need to learn the expected reward function. This is the only case that I am aware of where you need to learn an expected reward function, but I'd be interested to hear of other cases.

Answer 4

In reinforcement learning, the agent learns the value function, not the reward function. The value of an action is it's overall utility; for example, an action may bring a high reward, but lead to low-value states, making it low-value.

What the quote says is that sometimes the world is complicated: some problems have state-action spaces that are too large to enumerate, so we don't have explicit transition probability and reward functions. In such a case the agent would have to learn some generalization of the true value function.

Answer 5

I think those notes are slightly confusing. The rewards should be designed by you to encourage your RL agent to optimise the behaviour you want to see. The transitions between states are given by the enviroment and will often be estimated from data. I think what Andrew is refering to is a situation where from state A you could transition to state B which you give reward x or state C which you give reward y. If states B and C are identical except for the differences in reward, you would often eliminate states B and C and give state A a reward estimated from data which shows how often you transition to B or C from A.