# relationship between the Q value and the reward function in reinforcement learning

In the paper by Ng and Russell (2000), a section there discusses about the need to maximize the difference between the quality of the optimal action and the quality of the next best action.

The quality of the action is simply the $Q(s,a)$ function and the action of maximizing the difference between the first and second best is given by:

$$\max \left ( \sum \left ( Q_{\text{best}} - \max Q_\text{next best} \right) \right)$$

Putting it all together, the optimization problem becomes

maximize $\sum \min \{ (P_{a_1}(i) - P_{a}(i))(I - \gamma P_{a_1}(i))^{-1}R\} -\gamma |R|_1$

My question:

1. What happened to the $Q$ values? Why did it not appear in the optimization problem. I was so expecting it to appear since we went to the trouble of maximizing the difference between the first and second best values.
• Please improve the question by adding the title of the paper (and if possible, a link, the title is more important though to find it in case of link rot) :) Dec 21 '15 at 11:56

$$V^\pi=R+\gamma P_{a_1}V^\pi \therefore V^\pi=(I-\gamma P_{a_1})^{-1}R$$
$$Q^\pi=P_a V^\pi=P_a (I-\gamma P_{a_1})^{-1}R$$