In DDPG and DQN there are loss functions of the form below. But if we wait for a game/episode/rollout to complete, we can know the actual value for the future discounted rewards. Why not use that instead of relying on this recursion?
For reinforcement learning value function updates, you can use any valid estimate for return in place of the temporal difference target (TD target). Each type of estimate has different qualities that make it more or less useful depending on the problem you face.
The loss function you show is for a single-step Q learning update. The TD target value in it has the following useful traits:
It requires no knowledge of a model of the environment, you don't need access to the game rules. If you do have access to the game rules and want to use them, then this might not be a useful trait.
It requires no further adjustment for actions taken "off-policy". The loss is the same, and updates will converge towards optimal action values (for the optimal policy) even if behaviour is not optimal. There are some constraints on this - the behaviour policy must cover the optimal policy, and there is still sampling bias when using approximators such as neural networks.
It can be used to make immediate updates after each step, without waiting for an episode to end.
It has low variance, only random elements from a single time step are considered. A full trajectory to the end of an episode includes random elements from the policy and the environment for each time step, which can significantly increase the variance, depending on the environment.
There is one major negative trait:
- It is a biased due to initial arbitrary estimates. Actual samples of return, if taken on policy, or adjusted using importance sampling when off-policy, are not biased.
In practice, the variance and bias trade-off, plus the benefit of immediate online learning, can often favour using a TD target instead of full Monte Carlo sample. This is dependent on the problem.
You can mix and match the estimate mechanisms. A popular choice is to use eligibility traces to control a mix between single-step TD and Monte Carlo updates.
As another example, mixing things in a completely different way, I have successfully adapted the loss function in Connect 4 games, to use a full negamax search with optimal result when N steps away from the end (i.e. an N-ply search), falling back to single-step Q learning for the estimate when the search did not fully resolve the game.