17
$\begingroup$

While studying Reinforcement Learning, I have come across many forms of the reward function: $R(s,a)$, $R(s,a,s')$, and even a reward function that only depends on the current state. Having said that, I realized it is not very easy to 'make' or 'define' a reward function.

Here are my questions:

  1. Are there rules on how to make reward functions?
  2. Are there are other forms of the reward function? For example, a polynomial form perhaps that depends on the state?
$\endgroup$
21
+50
$\begingroup$

Reward functions describe how the agent "ought" to behave. In other words, they have "normative" content, stipulating what you want the agent to accomplish. For example, some rewarding state $s$ might represent the taste of food. Or perhaps, $(s,a)$ might represent the act of tasting the food. So, to the extent that the reward function determines what the agent's motivations are, yes, you have to make it up!

There are no absolute restrictions, but if your reward function is "better behaved", the the agent will learn better. Practically, this means speed of convergence, and not getting stuck in local minima. But further specifications will depend strongly on the species of reinforcement learning you are using. For example, is the state/action space continuous or discrete? Is the world or the action selection stochastic? Is reward continuously harvested, or only at the end?

One way to view the problem is that the reward function determines the hardness of the problem. For example, traditionally, we might specify a single state to be rewarded: $$ R(s_1)=1 $$ $$ R(s_{2..n})=0 $$ In this case, the problem to be solved is quite a hard one, compared to, say, $R(s_i)=1/i^2$, where there is a reward gradient over states. For hard problems, specifying more detail, e.g. $R(s,a)$ or $R(s,a,s^\prime)$ can help some algorithms by providing extra clues, but potentially at the expense of requiring more exploration. You might well need to include costs as negative terms in $R$ (e.g. energetic costs), to make the problem well-specified.

For the case of a continuous state space, if you want an agent to learn easily, the reward function should be continuous and differentiable. So polynomials can work well for many algorithms. Further, try to remove localised minima. There are a number of examples of how NOT to make a reward function -- like the Rastrigin function. Having said this, several RL algorithms (e.g. Boltzmann machines) are somewhat robust to these.

If you are using RL to solve a real-world problem, you will probably find that although finding the reward function is the hardest part of the problem, it is intimately tied up with how you specify the state space. For example, in a time-dependent problem, the distance to the goal often makes a poor reward function (e.g. in the mountain car problem). Such situations can be solved by using higher dimensional state spaces (hidden states or memory traces), or by hierarchical RL.

At an abstract level, unsupervised learning was supposed to obviate stipulating "right and wrong" performance. But we can see now that RL simply shifts the responsibility from the teacher/critic to the reward function. There is a less circular way to solve the problem: that is, to infer the best reward function. One method is called inverse RL or "apprenticeship learning", which generates a reward function that would reproduce observed behaviours. Finding the best reward function to reproduce a set of observations can also be implemented by MLE, Bayesian, or information theoretic methods - if you google for "inverse reinforcement learning".

$\endgroup$
  • $\begingroup$ Hi, why is the mountain car problem a time-dependent problem? $\endgroup$ – GoingMyWay Jul 20 '18 at 3:03
  • $\begingroup$ I suppose the mountain car problem is "time-dependent" in that the problem requires the network to provide the appropriate sequence of commands, or a policy that determines the sequence. The idea is that if you only treat "position" as your state, then the problem is difficult to solve - you need also to consider your velocity (or kinetic energy etc). That's really all I meant to imply, with regard to choosing your state space wisely in time-dependent problems. $\endgroup$ – Sanjay Manohar Jul 20 '18 at 17:32
  • $\begingroup$ @SanjayManohar I don't think the mountain car problem is "time-dependent", unless by time-dependent you're talking about introducing a finite time horizon. Position and velocity are sufficient. $\endgroup$ – user76284 Jul 20 '18 at 20:54
  • $\begingroup$ I think this answer mixes up reward and value functions. For instance it talks about "finding" a reward function, which might be something you do in inverse reinforcement learning, but not in RL used for control. Also, it talks about the need for reward function to be continuous and differentiable, and that is not only not required, it usually is not the case. You are far more likely to find simple +1 for success, or fixed -1 per time step taken in the literature, than to find some carefully constructed differentiable heuristic. $\endgroup$ – Neil Slater May 23 at 8:31
3
$\begingroup$

Designing reward functions is a hard problem indeed. Generally, sparse reward functions are easier to define (e.g., get +1 if you win the game, else 0). However, sparse rewards also slow down learning because the agent needs to take many actions before getting any reward. This problem is also known as the credit assignment problem.

Rather then having a table representation for rewards, you can use continuous functions as well (such as a polynomial). This is the case usually when state space and action space is continuous rather then discrete.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.