3
$\begingroup$

In one of the easy cases of inverse reinforcement learning, we try to derive the unknown reward function assuming that an optimal policy is known and MDP is completely known.

Let $P_{a_1}$ be the probability transition matrix from one state to another, following the optimal policy $\pi(s) = a_1$. Assuming $n$ states, then this matrix is $n \times n$. On the other hand, $P_a$ is the transition matrix of all other policies that are not $a_1$. Still $n \times n$.

If $I$ is the identity matrix and $\gamma$ is the reward discount factor (scalar), what is the dimension of $R$ in the follow rule? (from Algorithms for inverse reinforcement learning, by Ng and Russell, 2000)

$$(P_{a_1} - P_a)(I - \gamma P_{a_1})^{-1}R \geq 0 $$

Is $R$ a vector? or $n \times n$ matrix? And what would this mean?

|cite|improve this question
$\endgroup$
  • $\begingroup$ is this presumed to be a linear function in a linear system? $\endgroup$ – EngrStudent - Reinstate Monica Nov 30 '15 at 13:29
  • $\begingroup$ I actually am not sure how to answer your question. What do you mean by linear system? $\endgroup$ – cgo Nov 30 '15 at 13:31
  • $\begingroup$ Is $P_{a}$ constant? $\endgroup$ – EngrStudent - Reinstate Monica Dec 1 '15 at 4:28
1
$\begingroup$

As the paper of Ng and Russell (2000) indicates in section 2.1, the reinforcement function $R$, takes as input a state, and as output has the reward, a real number. Therefore $R$ should be a vector of $n$ items. The result of equation (4) of the paper:

$$(P_{a_1} - P_a)(I - \gamma P_{a_1})^{-1}R$$

therefore also is a vector of $n$ items.

Note that the reward function can also be defined with the parameters state ($s$), next state ($s'$), and action ($a$): $R_{ss'}^a$ as done by Sutto and Barto (1998, section 3.6).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Hi, thank you for your reply. I am not sure if you are familiar with the Q-learning example of a robot in a house with different rooms. The aim of the robot is to exit the house, and until then, the reward it gets is 0 for every move to a wrong room. Some rooms are connected say room 0 with room 4; room 4 with room 3; etc. In this case, the reward is $n \times n$. The column is the current state the robot is in, the row is the next possible state. I am asking this because for me, it would make sense that $R$ is $n \times n$, but I could not understand what it could mean when it is a vector. $\endgroup$ – cgo Nov 30 '15 at 15:31
  • $\begingroup$ When it is a vector you simply define the reward in function of its current state only: $R(s)$. The algorithm you use to calculate the value function will then probably need some iterations more to arrive to a same kind of policy as when you predefine the reward in function of the current and next state: $R(s,s')$. $\endgroup$ – agold Nov 30 '15 at 15:39
  • $\begingroup$ Alright, let me think about what you said. Thank you for your help. $\endgroup$ – cgo Nov 30 '15 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.