1
$\begingroup$

I am currently reading the "Neural Networks and learning Machines" by Simon Haykins book. In chapter 12 on policy iteration I don't really see the two differences between the cost-to-go function and the Q-factor:

$J^{\mu_n}(i)= c(i,\mu_n(i)) + \gamma \sum_{j=1}^N p_{ij}(\mu_n(i))J^{\mu_n}(j)$

$Q^{\mu_n}(i,a)= c(i,a) + \gamma \sum_{j=1}^N p_{ij}(a)J^{\mu_n}(j)$

Where $a = \mu_n(i)$ all according to the book.

To me both functions are really saying and doing the exact same thing. I understand that $a$ is an additional variable of $Q$ but in essence since there is an individual J/Q for each $\mu_n$ both functions just do the same thing.Can someone help me out on what I am missing here

$\endgroup$
1
$\begingroup$

When you make a choice under policy $\mu_n$ then indeed you might express that as $a = \mu_n(i)$.

The Q-factor does not pre-suppose you will make the same choice as the policy on the next step, it allows you to evaluate the expected future cost for taking any action (and from that point on following $\mu_n$).

Differences between $Q$ and $J$ therefore allow you to select a different $a$ in next iteration of the policy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.