I am new to this and I have been trying to understand the relationship between these two for a while (also referring to this question about RRL, but it seems to me RRL is a subset of DRL algorithms, because in my understanding DRL is combination of neural networks with reinforcement learning and RRL uses a neural net with a certain feedback loop (LSTM layer?), which qualifies it for the "definition" od DRL as well. Is my understanding correct? If not, can you please explain?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
it seems to me RRL is a subset of DRL algorithms, because in my understanding DRL is combination of neural networks with reinforcement learning and RRL uses a neural net with a certain feedback loop (LSTM layer?), which qualifies it for the "definition" od DRL as well.
I think this is essentially correct, but you are right to put definition in scare quotes here. The qualifier deep is not always strictly applied, and the categorisation of reinforcement learning (RL) algorithms is not a mathematical formalism, but an aid to navigating between multiple different approaches.
Many of the approaches in RL can be combined, or are on a continuum of options. It is important to not get too hung up on definitions or look for some official RL taxonomy to guide all discussion.
In the case of RRL, there is an important difference with DRL using non-recurrent networks, and that is how the agent deals with observations, where a single observation at any timestep does not cover important state data.
A classic example of this might be a frame from a video or game used as an observation. A single frame may not give enough information on predictable motion, whilst multiple frames may do. In that case you can choose either non-recurrent DRL and analyse multiple observations into a state representation (maybe by simply concatenating several frames), or you can use RRL with raw observation data per timestep. The RRL should be able to solve the state representation by learning useful internal features summarised from observation history.
The caveat is that RRL can be a little bit more sensitive to hyper-parameter choices, and does not allow so easily for tricks such as experience replay used to make e.g. DQN stable and more sample efficient.
