2
$\begingroup$

In Reinforcement Learning (RL), we specify the reward function so that the agent can learn the optimal policy. This reward function can come in various forms. It can be a scalar, a function, or anything else.

In the problem of Inverse Reinforcement Learning (IRL), the reverse happens: we are trying to find the reward function that can explain an expert's/demonstrator's behaviour.

My question is, upon finding this reward function, in what form does the reward function appear? Is it a scalar? A Function? or What?

I am trying to learn IRL, looking at some codes in the internet that demonstrate this concept. So far I am not really sure what the output is and how the reward looks like.

Your insights are appreciated.

$\endgroup$
  • $\begingroup$ Risking the obvious, the reward function is what it is in any MDP: A function mapping $(s,a,s')$ tuples to real numbers. What else are you getting at? $\endgroup$ – Sean Easter Feb 6 '17 at 19:59
  • $\begingroup$ I am entirely new to this. So in the forward RL problem, the output is a policy that an agent follows, which maximizes the expected cumulative rewards. So for example in a gridworld, I am able to see a path traced by an agent which gives maximum expectation of rewards. In IRL, will the output be just a bunch of numbers (the rewards)? Are there example codes in Matlab that demonstrates this idea? All I see are lectures without demonstrations/examples. It is quite hard to visualize. $\endgroup$ – cgo Feb 7 '17 at 2:06
  • $\begingroup$ Try my answer. To the question about code I can't help, but you may find code associated with publications by academic research groups that have studied IRL. $\endgroup$ – Sean Easter Feb 7 '17 at 13:53
1
$\begingroup$

It's worth noting two general facts of MDPs and IRL:

  • The reward function is a function mapping $S \times A \times S$ to the real line. (Many examples use only $s'$ to simplify exposition.)
  • This does not differ from reinforcement learning to inverse reinforcement learning: The goal of IRL is to produce a function that explains observed, optimal behavior.

Together, these two facts demonstrate that the form of the function output by IRL depends entirely on the state and action space. How should we visualize this function? That can get hairy pretty quickly.

Let's take a simple, deterministic grid world wherein reward depends only on the new state $s'$. You can think of the reward function for this problem as a set of rewards corresponding to state; visualize it as a heat map of the grid, with darker squares corresponding to higher rewards.

Now suppose that reward depends on the direction taken to arrive in the state. You'll now need four maps, each with a blank row or column. (One can't move rightward into the left-most square, e.g.)

Suppose finally that transitions are not deterministic, and that rewards depend on the full tuple $s,a,s'$. I'll leave this one to the interested, but suffice to say it depends on the transition function. (You'll need more maps.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.