How to train a stock trading neural network so that the 'profit' parameter is maximized?

Question

I am watching some beginner level video training on neural networks using Tensorflow / Keras to get a better understanding of how they work and how to best implement them.

I have some questions on how one feeds back signals to get the network to train itself. For example, lets say I am building a stock trading NN with I/O as follows:

Inputs:

• The last say 20 daily prices, volumes, moving averages, MACD signals or other financial indicators
• Signals that I supply, indicating where I would buy and sell

Output:

• A Buy signal and a Sell signal (or one signal that goes to 1 after buy and returns to 0 after sell)

Ideally the network would learn how to output buy and sell signals such that the difference between the buy and sell signals, ie. profit, is maximized. For example when price is trending upward it would issue a buy, and when it starts to slope down a sell occurs. The optimized network would adjust itself to buy at the lowest point and sell at the highest point possible.

My Question

I am not clear how the 'profit' signal is used to train the system. The profit is independent of the input and output signals, it is a function of both, and it is calculable only after a sell. How do I go about telling the NN to optimize itself to maximize an arbitrarily calculated value?

From my understanding this seems like an ideal application for NNs, Im just a bit unclear on how to proceed. Thanks for any advice.

---

And if I may ask a second question, let's say I wanted to run my NN against multiple stocks, each with its own unique behaviours; eg. it might be especially volatile or its median price may be 100 times the first stock.

In that case, would I create a whole new NN for each stock, or is it as simple as just adding a new input with a fixed constant for each different stock?

• StockA = 0.1
• StockB = 0.2
• StockC = 0.3

Would this cause training to apply a whole new set of weightings / biases for each stock?

Answer 1

More of an extended comment than of an explicit answer - but too long for comments:

1) A distinction needs to be made between prediction and optimization. Prediction is used to model phenomenon over which we have no control (i.e. we are merely a passive observer) - in your case this is the whether the stock will be trending up or trending down.

Optimization is used for variables over which we have control, i.e. we decide what that variable is (i.e. we interact with environment). In your case this whether to buy or sell.

There will be several scenarios for example where the stock is trending up, but buying isn't the best decision to take. At the very least, you should have three target values to train your network (1: Buy, 0: Wait, -1: Sell) - but even if you did that, your approach is still problematic because your conflating a prediction problem and a decision/optimization problem.

2) You might be better off using reinforcement learning instead of conventional neural networks. W/R to your problem, reinforcement learning has two advantages: In reinforcement learning, a model is trained to maximize a target function, as opposed to conventional neural networks where the model is trained to minimize a loss function. And reinforcement learning is better suited for problems where the goal is a long term goal, compared to supervised learning.

Answer 2

It's almost a truism to say that in any practical problem, it's not sufficient to throw neural networks at a problem and expect to strike it rich. Often, constructing better features is the harder, and most profitable, part of developing a better machine learning model.

Framed a different way, how do you plan to distill all relevant market data and present it to a neural network? Historical data is only a part of the puzzle, since, as we all know, past performance is no guarantee of future returns. A price bar for a security is just part of the picture. What do you make of the Fed's plan to raise interest rates in 2019? Also, it appears that a yield-curve inversion could be on the horizon, and that often precedes recessions. The same goes for declines in microchip demand. How does your model "price in" that information?

An example of using reinforcement learning for stock trades can be found in Maxim Lapan, Deep Reinforcement Learning Hands-On. The code for that chapter (and the rest of the book) is on Github.. (Full disclosure: I've made minor contributions to the repository.)

Reinforcement learning is an attractive way to frame the problem, since the core task of RL is to train an agent how to make optimal decisions (profit) when interacting with an environment (the stock market). Estimating discounted future rewards is a part of reinforcement learning objectives, and is the machine learning analogue to market gurus (Jim Cramer, Warren Buffet, whoever) making guesses predictions about market moves.

Additionally, RL is probably a better framework than supervised learning because labeling a financial time series and providing it to a model seems both labor-intensive and pointless. The goal isn't to build a model that reproduces a particular analyst's hindsight, but to build a model that can correctly estimate future return and act accordingly. In this scenario, we don't have the rewards data available - but by running experiments, the agent can (hopefully) learn optimal actions.

Naturally, it is hard to build an RL agent that is profitable. The agent proposed and tested in the book chapter fails to generalize: when the agent is applied to new data, it quickly loses all of its money. (A piece of life advise: if you decide to actually use a model to make real trades, only give it money that you can afford to lose!)

Marcos López de Prado's Advances in Financial Machine Learning (2018) is something of a how-to manual for the sorts of people who have a few million dollars lying around to fund a multi-year R&D effort. It outlines how using machine learning in finance is fundamentally distinct from funding active managers, and how the task requires a multi-disciplinary approach (such as high-performance computing specialists, machine learning experts, programmers, data engineers).

A large portion of Prado's book focuses on the problem of understanding how to measure model generalization, which is a more technical way to say "here's how to assess the risk that you will bankrupt the firm with your newfangled computer program."

---

It's easier to start learning about machine learning on simpler problems, or problems that we understand well. I have a political science degree; when I started doing machine learning in earnest, I would work on problems concerning elections data, because I understand the underlying material very well. Unless you consider yourself a finance expert, I'd start with a topic that's a little simpler. Or, at a minimum, I'd study the core elements of financial modeling, including ARIMA models, CAPM and Fama-French for a start. David Ruppert's Statistics and Data Analysis for Financial Engineering provides a good overview of these topics among others.

I also suggest reading a few posts at https://mathinvestor.org/ to better understand the kinds of hurdles that you are likely to face in this project.