Is reward function needed to be continuous in deep reinforcement learning? It should be noted that the reward is used for gradient computation

The algorithm I used is Proximal Policy Gradient(Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J]. arXiv preprint arXiv:1707.06347, 2017.)

The objective function is as follows:

enter image description here

The advantage is denoted as A^hat above, and it is derived from: enter image description here

And you can see that A is from state-action value and state value, which are derived from immediate reward.

  • $\begingroup$ Could you clarify what you mean by "the reward function"? Perhaps give one of the equations you mention, to make it clear which term you are calling that. For instance if you mean the term $r$ or $R$ or $R_t$ that might appear in e.g. $Q(S,A) \leftarrow Q(S,A) + \alpha(R + \gamma Q(S', A') - Q(S,A))$ for SARSA update, then typically that is not even considered to be a function (although no problem if you want to notate it as a function) $\endgroup$ – Neil Slater Dec 5 '17 at 8:16
  • $\begingroup$ reward is r in my question and when it is used in deep reinforcement learning, it may participate in gradient calculation. $\endgroup$ – mac wang Dec 5 '17 at 11:48
  • $\begingroup$ I need more to go on than that before I can answer. Please show the formula with the reward function in context. Use edit to add it to the question, so it is clear which function you are referring to. I am concerned that you have written "reward" but are trying to make a decision about some other related value - such as lambda return, value function or action-value function etc. The main reason I am concerned is that you have used the term "reward function" and typically reward is not expressed as a function (but there is no harm if it is, e.g. for practical purposes in code) $\endgroup$ – Neil Slater Dec 5 '17 at 12:48
  • $\begingroup$ Sorry, I have edited and appreciate your help. $\endgroup$ – mac wang Dec 5 '17 at 12:59
  • $\begingroup$ No problem. It's a lot better to see context. However, I don't see any "reward function" in the formulae you have added. Could you clarify which of them is the "reward function"? If you mean $\hat{A}$, that is the advantage estimation function, and must depend smoothly on parameters $\theta$ . . . typically you might write it as $\hat{A}(s, a, \theta)$ to show that it depends on $\theta$ $\endgroup$ – Neil Slater Dec 5 '17 at 13:01

Although you can define the reward as a "reward function", and you may have computer code that calculates reward from a function call with current state and action as inputs, typically reward is not considered a mathematical function. It is a variable that you can observe.

So to answer this, assuming you mean "reward" where you say "reward function":

Is reward function needed to be continuous in deep reinforcement learning? It should be noted that the reward is used for gradient computation

No, there is no requirement for reward to be drawn from any continuous function. That is because the value of $R_t$ is produced by the environment, independently of the parameters $\theta$ that the policy gradient is with respect to. Changing any part of $\theta$ would not change the value observed in the same context (although it may change whether you ever observe the same value again). In fact this is used when deriving your first equation in the Policy Gradient Theorem (see appendix 1 of this paper), the gradient of $r$ is assumed to be zero when expanding terms.

Intuitively, the reward is data that your algorithm learns from. It does not make sense to ask about the gradient of reward w.r.t. learnable parameters, any more than it makes sense to ask about gradient of input data from supervised learning w.r.t. learnable parameters*.

* In some contexts - e.g. style transfer for images - we do take gradients of input data in order to modify it - technically gradients of a loss function w.r.t. input, not input w.r.t. learnable parameters (that would still be zero). There are RL contexts where you fit reward structure to observed behaviour where this could be useful (e.g. inverse reinforcement learning), but that is not what you do when training an agent to optimise total reward in an environment.

  • $\begingroup$ Oh, the gradient of r in policy gradient is 0! Thank you! $\endgroup$ – mac wang Dec 5 '17 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.