1
$\begingroup$

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 323 (Policy gradient chapter) it says that:

"Perhaps the simplest advantage that policy parameterization may have over action-value parameterization is that the policy may be a simpler function to approximate."

Can anyone please explain the reason? Thank you

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Consider a game on the positive half of the number line, where you start at some integer $k$, and can move down by 1 or up by 1 each turn. The reward function is $f(x)$ for some monotonically decreasing function $f$ which is very hard to model -- for example, $f$ could be the negative cost of the optimal solution to a TSP problem on the first $x$ points in some set.

Then clearly, computing all these rewards and the action-value function is quite difficult. However, the optimal policy is simple: move down by 1 always.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Can you explain more everything after "-- for example", please? $\endgroup$
    – Ali Ghghgh
    Commented Dec 18, 2018 at 3:21
  • $\begingroup$ If you're unfamiliar with the traveling salesman problem, then that probably wasn't a helpful example. Just imagine that $f$ is a very hard function to model -- the world is filled with such functions. $\endgroup$
    – shimao
    Commented Dec 18, 2018 at 3:23
  • $\begingroup$ Ok, thanks. But how about moving down by 1 in the game. What was the goal of the game $\endgroup$
    – Ali Ghghgh
    Commented Dec 18, 2018 at 3:25
  • $\begingroup$ @AliGh the MDP formulation does not require a goal -- usually the problem is considered solved when the reward is maximized. In this case, reward is maximized when you reach $0$, since $f$ is maximized there. $\endgroup$
    – shimao
    Commented Dec 18, 2018 at 3:28
  • $\begingroup$ Actually, it is true when we have the reward function. Otherwise, we need to know the object. $\endgroup$
    – Ali Ghghgh
    Commented Dec 18, 2018 at 3:55
-1
$\begingroup$

Simple rule, action-value would be a mapping to the discrete function while policy network would be mapping to the continuous field. A continuous function is simpler to approximate.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm pretty sure this is not right. What do you even mean by " action-value would be a mapping to the discrete function " $\endgroup$
    – shimao
    Commented Dec 18, 2018 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.