3
$\begingroup$

In DDPG and DQN there are loss functions of the form below. But if we wait for a game/episode/rollout to complete, we can know the actual value for the future discounted rewards. Why not use that instead of relying on this recursion?

enter image description here

$\endgroup$
4
  • $\begingroup$ Could you provide references? Where did you get this loss definition? Also this question is not about DQN/DDPG exclusively rather more generic Q-learning question. Reformulation of the question might be needed. But in general future discounted rewards are not known and might change during exploration. $\endgroup$
    – msuzen
    Jul 10, 2021 at 4:03
  • $\begingroup$ The equation is DDPG. I think the original paper is paywalled, but the algorithm from the paper is in here: keras.io/examples/rl/ddpg_pendulum. Regarding your comment, can't I wait for a rollout to complete, then the discounted rewards can be known? The point about exploration is a good one though, that would indeed corrupt the rewards relative to the true policy. $\endgroup$
    – Mastiff
    Jul 11, 2021 at 17:15
  • $\begingroup$ Not sure if we can wait for roll out, because we are updating at each step. So depending on the path we follow we don't know yet how discounted rewards will turn out. $\endgroup$
    – msuzen
    Jul 11, 2021 at 19:07
  • $\begingroup$ Agreed, if you want to update each time step. The DDPG flow shows it this way, but I've had success with other algorithms waiting for each rollout to complete before training. I suppose it depends on the application (steps per rollout, among other things) how much it matters. $\endgroup$
    – Mastiff
    Jul 11, 2021 at 19:41

1 Answer 1

1
$\begingroup$

For reinforcement learning value function updates, you can use any valid estimate for return in place of the temporal difference target (TD target). Each type of estimate has different qualities that make it more or less useful depending on the problem you face.

The loss function you show is for a single-step Q learning update. The TD target value in it has the following useful traits:

  • It requires no knowledge of a model of the environment, you don't need access to the game rules. If you do have access to the game rules and want to use them, then this might not be a useful trait.

  • It requires no further adjustment for actions taken "off-policy". The loss is the same, and updates will converge towards optimal action values (for the optimal policy) even if behaviour is not optimal. There are some constraints on this - the behaviour policy must cover the optimal policy, and there is still sampling bias when using approximators such as neural networks.

  • It can be used to make immediate updates after each step, without waiting for an episode to end.

  • It has low variance, only random elements from a single time step are considered. A full trajectory to the end of an episode includes random elements from the policy and the environment for each time step, which can significantly increase the variance, depending on the environment.

There is one major negative trait:

  • It is a biased due to initial arbitrary estimates. Actual samples of return, if taken on policy, or adjusted using importance sampling when off-policy, are not biased.

In practice, the variance and bias trade-off, plus the benefit of immediate online learning, can often favour using a TD target instead of full Monte Carlo sample. This is dependent on the problem.

You can mix and match the estimate mechanisms. A popular choice is to use eligibility traces to control a mix between single-step TD and Monte Carlo updates.

As another example, mixing things in a completely different way, I have successfully adapted the loss function in Connect 4 games, to use a full negamax search with optimal result when N steps away from the end (i.e. an N-ply search), falling back to single-step Q learning for the estimate when the search did not fully resolve the game.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.