20
$\begingroup$

I recently came across the word of "Recurrent Reinforcement Learning". I understand what "Recurrent Neural Network" is and what "Reinforcement Learning" is, but couldn't find much information about what a "Recurrent Reinforcement Learning" is.

Can someone explain to me what is a "Recurrent Reinforcement learning" and what is the difference between "Recurrent Reinforcement learning" and normal "Reinforcement learning" like Q-Learning algorithm.

$\endgroup$
17
$\begingroup$

What is a "recurrent reinforcement learning"?

Recurrent reinforcement learning (RRL) was first introduced for training neural network trading systems in 1996. "Recurrent" means that previous output is fed into the model as a part of input. It was soon extended to trading in a FX market.

The RRL technique has been found to be a successful machine learning technique for building financial trading systems.

What is the difference between "recurrent reinforcement learning" and normal "reinforcement learning" (like Q-Learning algorithm)?

The RRL approach differs clearly from dynamic programming and reinforcement algorithms such as TD-learning and Q-learning, which attempt to estimate a value function for the control problem.

The RRL framework allows to create the simple and elegant problem representation, avoids Bellman's curse of dimensionality and offers compelling advantages in efficiency:

RRL produces real valued actions (portfolio weights) naturally without resorting to the discretization method in the Q-learning.

RRL has more stable performance compared to the Q-learning when exposed to noisy datasets. Q-learning algorithm is more sensitive to the value function selection (perhaps) due to the recursive property of dynamic optimization, while RRL algorithm is more flexible in choosing objective function and saving computational time.

With RRL, trading systems can be optimized by maximizing performance functions, $U( )$, such as "profit" (return after transaction costs), "wealth", utility functions of wealth or risk-adjusted performance ratios like the "sharpe ratio".

Here you will find a Matlab implementation of the RRL algorithm.


References

Reinforcement Learning for Trading

Reinforcement Learning for Trading Systems and Portfolios

FX trading via recurrent reinforcement learning

Stock Trading with Recurrent Reinforcement Learning (RRL)

Algorithm Trading using Q-Learning and Recurrent Reinforcement Learning

EXPLORING ALGORITHMS FOR AUTOMATED FX TRADING – CONSTRUCTING A HYBRID MODEL

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ @AntonDanilov I'm not sure if you're aware of this. The guy who came up with this idea (your first ref., J Moody) has been running a fund using this algo -- and his performance has been far from spectacular. $\endgroup$ – horaceT Jan 1 '18 at 19:51
  • $\begingroup$ So, good to know but how it does changemy answer $\endgroup$ – Anton Danilov Jan 3 '18 at 7:30
2
$\begingroup$

The distinction of (Deep) Recurrent RL, is that the function mapping the agents observations to its output action is a Recurrent Neural Network.

A Recurrent Neural Network is a type of neural network that processes each observation sequentially, in the same way for each time step.

Original paper: Deep Recurrent Q-Learning for Partially Observable MDPs

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.