# Is it a valid claim, that by differencing a time series, it loses its memory, and as a result its predictive power?

Marcos Lopez de Prado seems to be a well known and renowned machine learning expert in the field of finance. I am very far from his level, as have not yet finished my PhD in economics, and only have an applied level statistical knowledge. I have encountered a much cited paper of Lopez de Prado. I can not say I completely understand all the mathematical parts. But there are some claims in the paper, which seem to completely contradict things, which I have learned on statistics and economics thus far, or at least seem to be illogical for me. For a specific example, the paper under the section Pitfall #4 and Solution #4 suggests, that by differencing time series to make them stationary for classical statistical models (ARIMA etc.) removes the memory of the series and thus makes them lose the predictive power:

The conclusion is that, for decades, most empirical studies have worked with series where memory has been unnecessarily wiped-out. The reason this is a dangerous practice is that fitting a memory-less series will likely lead to a spurious pattern, a false discovery. Incidentally, this over-differentiation of time series may explain why the Efficient Markets Hypothesis is still so prevalent among academic circles: Without memory, series will not be predictive, and researchers may draw the false conclusion that markets are unpredictable.

In economics there really is a simplified theoretical model on equity returns, which posits, that it is a memory-less white noise, and the prices (the integrated returns) follow a random walk. But from an empirical perspective, as far as I understand the memory-less attribute of returns pertains only to the individual data points themselves, not the series as a whole. A differentiated series still should have a "collective" memory put together, and it has almost the same information as the integrated version, only lacks a constant value. So it should have the same predictive power as well, should it not? Or it is me, who has a lack of understanding?

• BTW (+1) Good question. – Carl Jul 6 '19 at 20:06

This tries to answer the original question and not get into Marcos's paper etc. If you think that the level of a variable ( say log price ) has information, then differencing the series ( to obtain returns ), throws out information. If you don't think that the level has information, then differencing is fine. Engle and Granger in their 1987 econometric paper showed how it is possible to consider both levels and changes in the relationship between two series ( X and Y) through the use of an ECM. But it does not mean that there can't be cases where one doesn't care about the levels and is only interested in changes ( or vice versa ).

On a different note, here's a piece of advice. Whenever you read anything about strategies and techniques and approaches in finance don't put too much weight in them because, if the author REALLY TRULY HAS SOMETHING THAT WORKS, he-she is not going to divulge it anyway. Most of the stuff you read will be purposely vague and general and unless you know the details of what the person actually does, not terribly useful. That's not to say that Marcos doesn't write interesting papers but he's not going to tell you what he actually does so it's best to read his or anyone's presentations with that in mind.

• +1. My intuition is that differencing is a stronger statement than "doesn't contain information" but rather is asking the model to average across what might be different regimes (high/low/etc), integrating out a nuisance variable, as it were. – Wayne Jul 6 '19 at 19:50
• Hi Wayne: To be honest, I'm not sure what Marcos is referring to when he says memory is wiped out. Possibly he's referring to where people use rolling windows so that information in the far past is ignored ? – mlofton Jul 7 '19 at 4:50
• I think stationarity wipes memory by making snippets of the time series appear the same (in some summary, at least). So your process could've experienced a low regimine and then a high one, but that history (memory) is wiped out by differencing. It may have also acted differently under those two regimes, but your ARMA model will be stuck averaging across them, oblivious to the fact they existed. – Wayne Jul 7 '19 at 12:10
• gotcha. yes, regime changing also makes things difficult. – mlofton Jul 8 '19 at 14:13

The signal processing approach to such problems may be easier to understand. Among other things, the signal to noise ratio (SNR) for stocks may be so high as to render signal detection difficult, there are none-the-less many off-the-shelf algorithms for improving SNR, and that is perhaps of more interest to you. Such algorithms may sacrifice some signal (lossy algorithms) to accomplish noise reduction. The simplest algorithm to understand is perhaps addition. For example, if the trends are low frequency (long term) then reduction of the higher frequency noise by averaging (read as addition of) adjacent temporal signals or averaging over a short time 'window' will tend to reduce high frequency noise more so than low frequency signal thereby improving SNR. In the frequency domain this is called a low pass filter.

With respect to the question, addition and subtraction reduce information in the sense that when an average or difference is created, one loses the information as to what the numbers were before combining them. For example, given a $$5$$ as a result that could have come from $$6-1$$, $$10-5$$ or an uncountably infinite number of other number pairs.

Regarding information content such things are context dependent. If we assume that the noise contains no information, and that only the signal has information content, the goal of signal processing would then be to isolate the signal with the least possible signal loss while reducing the memoryless noise to zero. On the other hand, if the context is that both signal and noise contain information, then obviously noise reduction would seriously reduce information content.

With respect to prediction, if the noise contains no information then one could establish bounds or an 'envelope' in which the noise is contained. That is not exactly a prediction in the same sense as $$y=f(t)$$, but when coupled with the low pass signal may none-the-less offer a prediction range.

In the real world (assuming money is real) the high frequency data contains less information (but still some) and the low frequency data relatively more information. Still, signal processing may be useful. For example, examining the data in the frequency domain may reveal repetitive occurrences that are not obvious in the data itself, and such a process (Fourier transform) is lossless.

• I understand the approach you are saying. But when we are doing the differencing, I think, that at that point everything can be considered "useful" information and not noise. Only the estimation method, for which we feed the series - let it be a regression or machine learning based - will decide what is noise and what is useful information. Until that point I would say, that we need every piece of information we can feed into. – Betelgeux Jul 5 '19 at 13:06
• That is why it is hard for me the grasp the concept, of loosing information by differencing. If we difference, we can easily recreate the original series by integrating, only we are missing the initial value, or the offset, which I think @mlofton is also referring to. But I do not think that the lack of that one constant can hinder the predictive power of a model, especially if we are thinking on stock price series - where the absolute magnitude of a stock is not much of interest to anyone. – Betelgeux Jul 5 '19 at 13:07
• An other explanation I can think beside this, that maybe people are implying, that we are loosing information because of a floating point error when calculating the difference d series as a percentage change or logarithmic return. But that is much more a computer science issue, than a theoretical statistical one, and I am sure we can have enough computational capacity, that can reduce it to being marginal. – Betelgeux Jul 5 '19 at 13:07
• Losing information (spelling, common mistake, e.g., loose screw). A linear difference equation's information loss would, I suppose, depend on exactly what was done to produce it. In the particular case you are referring to, I do not know how much information would be lost; that depends on the exact algorithm used. – Carl Jul 5 '19 at 22:16

Since my similar question was flagged as duplicate (good debate in the comments!), I came across Simon Kuttruf's explanation on Medium:

for integer orders of differencing only a (small) finite set of past values is reflected in the resulting differenced series: the preceding value in first order differencing, two preceding values for second order differencing etc. While for fractional orders of differencing, all coefficients take on (asymptotically small) non-zero values and so past values get mixed into the differenced series, up to some chosen cutoff. This phenomenon is here referred to as ‘long memory’ (or ‘wipe-out of memory’ resp).

If the fraction $$d = .5$$, then the first four coefficients (check my calculations) according to the recursive formula (see Simon's article)

$$w_k = -w_{k - 1} \left( \frac{d - k + 1}{k} \right),$$ the first four values would be 1, -.5, -1/8, and -1/16, leading to the transformation

$$z_t = y_t - \frac{1}{2} y_{t - 1} - \frac{1}{8} y_{t - 2} - \frac{1}{16} y_{t - 3} + \ldots$$

According to the terminology, the "memory is preserved" because the coefficients are positive for infinitely many past values of $$y_{t-k}$$.

The one example I can think of where memory is clearly lost by a first difference but without any apparent consequences is a true random walk. Where $$\epsilon_i$$ are i.i.d. $$N(0, 1)$$ random variables, let \begin{align} y_1 &= \epsilon_1, \\ y_2 &= \epsilon_1 +\epsilon_2, \\ y_3 &= \epsilon_1 + \epsilon_2 + \epsilon_3, \\ y_4 &= \epsilon_1 +\epsilon_2 + \epsilon_3 +\epsilon_4. \end{align} Then the first difference $$y_4 - y_3 = \epsilon_4$$ losses all of $$y_4$$'s information about $$\epsilon_1, \ldots, \epsilon_3$$, whereas the fractional differencing transformation would contain a bit of all of the epsilons. Not that they'd help you to forecast.

Another maybe more profound example where "memory" matters is a conintegrated time series regression problem described in page 11 of these notes by Eric Sims. There, taking first differences of two cointegrated random walks results in bias in the estimation stage, because the error in the difference vs difference regression includes a correction term to bring things in line with the long run relationship. If you rid yourself of that long term cointegrating relationship by differencing, you'll suffer bias.

It's a well-know trade off between explanatory and predictive modeling.

We don't really do the difference nowadays but when you try to fit a model, the goal at the beginning is very important.

I can have the best predictive model of all time with non sense coefficient estimates or have a really good explanatory model that predicts poorly.

So basically when you do a differenciation of your series, you are more in a explanatory case than predictive so you sacrifice a little of the predictive power to have a better model in quality. Or at least you are ready to sacrifice but i don't think it can be proved that it will always make loose predictive power, maybe 95% of the cases base on personal experience.

• If you have terrible predictive power, why do you believe your explanation? – Dave Jul 3 '19 at 18:53
• When I said poorly, I should have said worse than an other modeling option with less explanatory power. – josef_joestarr Jul 3 '19 at 19:25
• @Betelgeux: I'm not clear on what you mean by the lack of one constant in your comment ? In time-series, differencing means that you know longer have the levels of the series at any point in time because you are differencing adjacent values. so, you lose a lot of information. But, particularly in finance, differencing can often make a series more stationary so it can be a useful thing. In machine learning, you could be correct in that it's less harmful to keep everything but time-series is pretty different. Note that you only lose one constant if the series had a constant level. – mlofton Jul 6 '19 at 16:07