Using Trend as a feature in time series sliding window?

Question

I have a time series, and i am using overlapping sliding window to extract features from each window and label it accordingly. In this Overlapping window of size n, i want to extract trend (linear fit) and use as a feature, any ideas how can i use it, for now i am taking slope coefficient and intercept as a feature from each window.
is taking slope and intercept of linear fit in a window as feature is okay, or are there any other techniques i can use? Because my goal is to consider trend of each window in to DL model.

Please give your suggestions and ask for more clarification if required.

UPDATE:

The blue box is window_1 (blue box), i am extracting features for each window and labeling them, as we can see the linear trend (approx, red line) in this window is slightly upward, and in window_2 (violet box) it is downward, i want to put this information in terms of features, how to do that? is taking coefficients of slope and intercept sufficient or something else i should do?
So in Deep Learning model entire window will be one observation.

Thanks, and i hope it should be more clear now.

Answer 1

We often de-trend the time-series before modeling other aspects of it. Depending on your needs, you can use differencing, i.e. model $y'_t = y_t - y_{t-m}$, or subtract the trend, e.g. using linear regression $y_t = \beta_0 + T_t + \varepsilon_t$ ($\beta_0$ is intercept and $T_t = \beta_1 t$ is the parameter for time), next subtract the estimated trend $\tilde{y}_t = y_t - (\beta_0 + T_t) $ and then model the $\tilde{y}_t$ using another algorithm. Notice that second approach is a direct consequence of assuming additive trend, e.g. $y_t = T_t + f(X_t) + \varepsilon_t$.^† When subtracting the trend $T_t$, it does not have to be linear trend, it can be smooth function (e.g. LOESS), piecewise-linear, moving average, but also averaging linear trends in overlapping windows, or something else.

Beyond simplicity, when modeling de-trended series you also easily visualize the series to gain better insights (what do you see when plotting de-trended series?) or notice problems with the assumed trend (e.g. "jumps" at the borders of windows, when using piece-wise trend components). The drawback of using two-stage approach, is that when estimating trend and other components using a single model, you would be also correcting for changes related to the other components, rather then ignoring them as with estimating the trend independently.

I don't see how using the slope parameters as features would have any added effect over de-trending the series. If you did so, your network would still need to learn how to use this information, so why not use model that learns the trend by itself (by adding time as a feature)?

^{† - like in in STL decomposition, $y_t = T_t + S_t + \varepsilon_t$, where $T_t$ is trend and $S_t$ is seasonality and $\varepsilon_t$ is random noise, or in Prophet, $y_t = T_t + S_t + H_t + \varepsilon_t$, where $T_t$ is piecewice-liear trend, $S_t$ is seasonality and $H_t$ are holiday effects (Taylor and Letham, 2017).}

Answer 2

The main thing to be careful of in predictive time series models is to avoid leaks from the future. The classic example is where you calculate the moving average or slope centered on the current point. This will include N points to the left and right, and the points to the right are from the future. You won't have those points when your model is actually going to be used.

So you need to think exactly what information your model will have at the point you want to make a prediction and not build your model on any other data. (That's why people use trailing statistics, though moving averages never really make sense, even if you use the less-wrong trailing version.)

In answer to your comment, I am uncomfortable with your proposed method for several reasons, but first let me say that I think your instinct on this is probably good: you don't want to come up with an ad hoc method for formalizing what you see in time series as you look at them. None of this "Well, if I subtract the 3rd, 5th, and 7th values from the average and take the largest...", and that's a good instinct.

But your method introduces other complications that worry me:

1. You’ll probably end up using a regression method which is thrown off by outliers. In that case a single spike point can cause it to calculate a misleading slope.

2. Do you understand regression and it’s disgnostics? Are you going to grab the slope and intercept coefficients without looking at their “statistical significance”? If not, what will you do if they are not significant?

3. If you do understand regression diagnostics, do you know how they can be inaccurate (overly optimistic) when applied to time series, and how to correct for that?

   Calculating trends by regression is trotting out machinery that is often more complex than people realize and yet not as powerful as State Space approaches, for example, that could give you direct answers.

   That’s why I’m encouraging you to think of other features that might be simpler and more directly address your actual targets.

4. Are you sure that there really are up and down segments? From what you say there's no overall trend or any seasonality, so the series is basically wandering around. Depending on the autocorrelation, this can make series that look like noise or that look like something's going on. But it really isn't. Using a complex tool like regression could fool you into thinking what you're seeing must be real, when it might just be noise.

   If you're using R, try: plot (arima.sim (n=50, list (ar=c(0.882)))) several times in a row. You'll see fascinating patterns, but...

On the other hand, if you are rigorous with training and testing and avoiding future leaks, you can certainly try the regression slope method and not care that much if it’s an approved idea or not: the proof is in the predicting. If it predicts well — and you’re not messing something up to fool yourself about your out-of-sample accuracy — it’s good enough. You might not be able to reverse-engineer your model for goodvexplanations of why, but if that’s not your concern, no need to worry. (And it might be slow to deploy on a large scale, but that’s another matter.)

Pre-edit continuation of answer:

Also, we love to calculate trends by throwing a set of data into a OLS regression. But is that actually necessary? Is the trend in this case just the first value in the window subtracted from the last? Or do you care about the number of positive point-to-point deltas versus negative deltas instead?

Last, you can't just use an OLS regression naively on a time series. Time series are autocorrelated, which violates an assumption of OLS. (The main effect is that your diagnostics are too optimistic and you think your slope is statistically different from 0 and it's not.) Not to mention that time series have seasonal effects. That's why there's old-school ARIMA. (A more modern approach is State Space, also known as Kalman, but that's probably too far afield for your simple application, hence my question about whether you even really need to do regressions at all.)

Answer 3

An alternative, perhaps more principled way, of estimating "local" trends is to use structural models "à la Harvey". These are models which can be cast in state-space form and allow explicit consideration of trend and seasonality among other things. They can be supplemented with fixed effects and be made more elaborate as needed.

Software is widely available: in R, function StructTS fits several different structural models, and there are packages for multivariate series as well.

Aside from the books by Harvey, Durbin and Koopman(2011), Time Series Analysis by State Space Methods, Oxford Univ Press, is a very nice introduction.

Answer 4

I cannot comment. Hence posting as an answer.

If you are not using Deep Learning, Is it mandatory for you to use overlapping sliding window? This technique is generally used in Deep Learning Sequence models where you convert your time series data into 'n' small sequences making them as 'n' training examples.

For standard Time series solution using ARIMA, you can use Tim's method.