Forecasting Prices vs Returns by Deep Learning

The question: It seems that (univariate) forecasting stock market done by websites using DL and LSTM actually does not work that well if we focus on returns instead of prices. What is a relatively quick fix for that?(or most important fix)

Explanation: There are hundreds of python DL codes in internet trying to forecast stock market(usually S&P 500 ) prices using LSTM and other methods mostly in keras/tensorflow(an example here:[1]). When I applied and graphed visually it seems to be a good forecast. On the other hand when we do the same method for "returns" , everything falls apart. Even a constant forecast (For example always predicating that the return will be 0.01 monthly) does better or relatively the same as DL. What is the reason and mostly importantly cure for that?

• I disagree with the answers below, both in conclusion, and motivation. I watched "think or swim" and paper trading sites, and it seems to me (personally) that owners thereof use them to adaptively extract patterns of investing by individuals. The non-long market tries very hard to be a zero sum game (Poundstone,2005, p389/827), which means it isn't about value creation but value extraction from the "non-winners". Buffets non-transitive dice show how with money and nerd you can lock others out of the "winner-pool". The actual system is the national economy which doesn't move fast, so glass. Nov 24, 2020 at 20:29
• If you have a good predictor for stock prices (which is an extraordinary statement), why not forecast the price and then calculate the forecasted return based on it (i.e. $p_t/p_{t-1}-1$)? You wouldn't need a separate model for returns (or am I missing something?). If it doesn't work well, obviously the stock price forecast wast not satisfactory to begin with (a pretty well established fact in financial mathematics). Nov 24, 2020 at 22:05
• @PaulG Forecasting the price and forecasting the return turn out to be quite different. "When I applied and graphed visually [price] seems to be a good forecast. On the other hand when we do the same method for "returns" , everything falls apart"
– Dave
Nov 24, 2020 at 22:16
• I was thinking something along the lines of: if we have a perfect price forecast, it will lead to the perfect return forecast since one follows from the other. You would only need to fit 1 model for the prices. In econometrics/finance returns are not forecasted themselves, since they change drastically (from positive to negative) in very short periods of time, making a return plot look like a dense zig-zag scatter. This is why there are models for their volatility instead (periods of larger vs smaller zig-zags) such as ARCH or GARCH. Nov 24, 2020 at 22:31
• So the "cure" would be simple: model returns with a different model or model their volatility only, since they're pretty much white noise (i.e. can't be forecasted in the mean precisely). Nov 24, 2020 at 22:37

What you've outlined is probably the single most common error that machine learning researchers make when analyzing financial data: it's trivial to discover that a great predictor of tomorrow's price is today's price.

The statistical term of art for this phenomenon is "non-." We have a number of questions about how to test for the stationarity of a time series. One such thread is How to know if a time series is stationary or non-stationary? In the particular case of time series analysis of financial data, it might be helpful to review a high-quality statistical text, such as Statistics and Data Analysis for Financial Engineering, Second Edition (David Ruppert & David S. Matteson). On page 308, we find the remark

As mentioned, many financial time series do not exhibit stationarity, but often the changes in them, perhaps after applying a log transformation, are approximately stationary.

(This is a quite extensive textbook about time series data and financial data, so it's worth reading in some detail if you're interested in how to pursue this project further.)

So to answer your question, the example neural networks that you mention discover that the financial data are non-stationary, and these models make use of that fact when making predictions. But if you look at returns, then the non-stationarity phenomenon disappears, and the model is not able to discover such a simple rule to exploit.

The cure, in some sense, is to discover what drives stock prices, either generally or in the specific case of the equities you're studying. The price changes every second -- why is that? What information could a person have that causes a 0.1% shift from minute to minute, or 1% day to day? It's unlikely that yesterday's price movement, or the price movement the day before, will tell you much of anything about tomorrow's price movement by itself with a high degree of precision -- because, as we know, past performance is no guarantee of future returns.

Framed in this way, the problem is not about choosing a certain kind of neural network, but instead making a neural network that has relevant data to inform its predictions. So, right now, you know that a good predictor of price tomorrow is the price today. To improve on that, you'll have to find timely information that improves upon the "best guess" provided by yesterday's price data.

As an example of what form this information might take, consider pairs trading. In the 1980s, Morgan Stanley quants invented "pairs trading" and the strategy was profitable for a while. The premise is that two highly correlated stocks will tend to move together, so if there is movement in one that's not present in the other, you can make a trade with thesis that eventually the two stocks will return to their equilibrium. So your neural network would use information about one stock to place trades on the second stock, and vice-versa. Naturally, pairs trading is only profitable as long as the premise that the pairs are strongly correlated is true.

• Comments are not for extended discussion; this conversation has been moved to chat.
– Sycorax
Dec 6, 2021 at 22:29

The question: It seems that (univariate) forecasting stock market done by websites using DL and LSTM actually does not work that well if we focus on returns instead of prices. What is a relatively quick fix for that?(or most important fix)

The motivation is quite simple. You can find it in any financial economic/econometrics text. As starting point we can consider stock price (log-price) as described from a Random Walk model (RW):

$$p_t = p_{t-1} + \epsilon_t$$

where $$\epsilon_t$$ are iid gaussian noise. Then $$E[p_{t+1}|I_t]= p_{t}$$

$$I_t$$ stand for information set at time $$t$$, but it boil down into $$p_t$$ in this model. So for log return = $$r_t = p_t – p_{t-1}$$ we have $$E[r_{t+1}|I_t]=0$$ for all $$t$$

Take away: price is non-stationary series, the best predictor for price in the future is price now, returns are stationary and best predictor for them is zero.

Candidate predictive model have to predict better than RW, using in some way $$I_t$$. usually is better to try with return than price, because the former are stationary (wide sense).

Under usual definition of predictability, price are quite well predictable while return are hard to predict. So your evidence is usual.

Machine learning can be used fruitfully in finance too, but beat RW is far from simple.

NOTE: Just to note that some comments below touch interesting points as equilibrium or others; however here we have to stay focused on prediction only. Others comment ask empirical support. About the last request, the story above is a model (theory) and, apart any details, it represent the oldest stochastic model in finance (Bachelier model: https://en.wikipedia.org/wiki/Random_walk#Applications; https://en.wikipedia.org/wiki/Bachelier_model ). RW became a benchmark model in stock prediction (and for other assets too). During the decades was proposed, in academy and financial industry, hundreds of predictive models. Models that, looking for predictability, refute the RW condition like $$E[r_{t+1}|r_t, r_{t-1} …]=0$$. Them searched some model for which $$E[r_{t+1}|I_t] \neq 0$$ and, then, to beat the RW in MSE or other metrics. However until now no one of them can beat the RW for all dataset (all country/index, all stock/name, all data frequency, ecc). So, it is useless to cite here specific articles that confirm or refuse returns predictability, this is an endless empirical debate. Moreover, for some RW version we can relax gaussianity and independence with weaker conditions but the basic implications above hold yet. Finally, the history seems me the strongest as possible empirical confirm that the basic RW implications above, in some extent, is consistent with data. Therefore it remain the more simple and more convincing way for explain why stock price are easily predictable while returns are very hard to predict.

Just another point. Some trader/analyst want to predict directly the prices and not the returns. Them intend this work as predict the direction of price in the next period and not the level (it is easy). These guys use something like “prices paths”. This target is not different that predict returns direction. However these guys do not realize that there are no way for infer something from non stationary and non ergodic time series.

• I agree that a RW process with 0-mean noise implies that such a sequence generated by such a process has expected return of 0. However, this observation does not, on its own, establish that stock prices are examples of a RW with 0-mean noise. In other words, why do we know that financial data conforms to the properties of this mathematical model?
– Sycorax
Nov 24, 2020 at 20:53
• Without some sort of positive drift it makes all stock investing idiotic doesn’t it? Nov 24, 2020 at 21:43
• If markets are efficient today is the best predictor of tomorrow. If someone could improve on it they would get rich and certainly not share their secret on a public forum. That said some make large amounts of money investing which is hard to attribute just to luck. Nov 24, 2020 at 23:08
• @user54285 Continuing your argument, if it were possible to predict future price changes profitably then that person who could do it would have done so, moving the current price towards the future price and eliminating the predictable future change. This is what makes markets efficient and prevents others from doing something similar. Nov 25, 2020 at 10:35
• @Henry I think my quibble with that is the mechanism. It is unlikely that a single investor could move the market. I think the argument for efficient markets would argue that this special knowledge would soon be known to all and that would move the market. Nov 25, 2020 at 16:08

Advances in Financial Machine Learning is a good reference for practical usage of ML in the context of financial time series.

Basically :

1. Formulating your label in term of level attained in a given amount of time (see chapter 3 barrier method) will help you build practical and realistic strategies. Unless you are doing some market making, you usually don't care about the price at the next ticker. For application you might need to go further and use meta labeling (ie. two algo, one to predict the trend, one to predict the amount to bet).

2. Regarding feature building, you often go with stationnary processes, that you obtain trough fractionnal differenciation (getting rid of noise without getting rid of information), instead of integer differenciation (0: price, 1: return). See Chapter 5 - Fractionnaly differentiated features.

3. As mentionned by others, there are complex mechanisms at play, and numerous actors on the market such that any meaningfull price prediction will be taken advantage of and nearly immediatly corected. That's why whitout external/original/new info, you won't get meaningfull prediction (and why underlying are often modelled as randow walks with drift for option pricing). Basically you need external info to predict price movement. See here for an exemple where tweets are used (financial-tweets).

So generally speaking, when using ML for Finance you just don't predict next price from past prices. There is barely any value in doing that. The exception might be in HFT when you are trying to predict price change from limit order book, but that is a very specific case. And, to answer your general question, there is no easy way to deal with it : any 'easy' solution has already been implemented and has been optimised to the point it is difficult to compete.