In the paper by Ng and Russell (2000), a section there discusses about the need to maximize the difference between the quality of the optimal action and the quality of the next best action.

The quality of the action is simply the $Q(s,a)$ function and the action of maximizing the difference between the first and second best is given by:

$$ \max \left ( \sum \left ( Q_{\text{best}} - \max Q_\text{next best} \right) \right) $$

Putting it all together, the optimization problem becomes

maximize $\sum \min \{ (P_{a_1}(i) - P_{a}(i))(I - \gamma P_{a_1}(i))^{-1}R\} -\gamma |R|_1$

My question:

  1. What happened to the $Q$ values? Why did it not appear in the optimization problem. I was so expecting it to appear since we went to the trouble of maximizing the difference between the first and second best values.
  • 1
    $\begingroup$ Please improve the question by adding the title of the paper (and if possible, a link, the title is more important though to find it in case of link rot) :) $\endgroup$
    – mlwida
    Dec 21 '15 at 11:56

The Q values are there. Section 3.1 of the paper, "characterization of the solution set," shows how to express the V and Q definitions in non-recursive linear form.

$$ V^\pi=R+\gamma P_{a_1}V^\pi \therefore V^\pi=(I-\gamma P_{a_1})^{-1}R $$

and, if I understood correctly,

$$ Q^\pi=P_a V^\pi=P_a (I-\gamma P_{a_1})^{-1}R $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.