7
$\begingroup$

I am familiar with "regular" cross-validation, but now I want to make timeseries predictions while using cross-validation with a simple linear regression function. I write down a simple example, to help clarify my two questions: one about the train/test split, one question about how to train/test for models when the aim is to predict for different n, with n the steps of prediction, in advance.

(1) The data

Assume I have data for timepoints 1,...,10 as follows:

timeseries = [0.5,0.3,10,4,5,6,1,0.4,0.1,0.9]

(2) Transforming the data into a format useful for supervised learning

As far as I understand, we can use "lags", i.e. shifts in the data to create a dataset suited for supervised learning:

input = [NaN,0.5,0.3,10,4,5,6,1,0.4,0.1]
output/response = [0.5,0.3,10,4,5,6,1,0.4,0.1,0.9]

Here I have simply shifted the timeseries by one for creating the output vector. As far as I understand, I could now use input as the input for a linear regression model, and output for the response (the NaN could be approximated our replaced with a random value).

(3) Question 1: Cross-validation ("backtesting")

Say I want to do now 2-splits, do I have to shift the train as well as the test sets?

I.e. something like:

Train-set:

Independent variable: [NaN,0.5,0.3,10,4,5]

Output/response variable:[0.5,0.3,10,4,5,6]

Test-set:

Independent variable: [1,0.4,0.1]

Output/response variable:[0.4,0.1,0.9]

(ii) Question 2: Predicting different lags in advance:

As obvious, I have shifted dependent to independent variables by 1. Assuming now I would like to train a model which can predict 5 time steps in advance -- can I keep this lag of one, and nevertheless use the model to predict n+1,...,n+5,... or do I change the shift from independent to dependent variable to 5? What exactly is the difference?

$\endgroup$
  • 2
    $\begingroup$ Traditionally, you would cross-validate time series models by using a rolling window within the sample. But more recently some other approaches have been developed, too; see Rob J. Hyndman's blog post "Cross-validation for time series". By the way, your question may be a duplicate of one of the earlier questions on time series cross validation – see these. $\endgroup$ – Richard Hardy May 7 '18 at 12:41
  • $\begingroup$ I understand that, but the rolling window is used in both A and B. My question is rather if i would need for both train and test sets and input and a (shifted) output vector $\endgroup$ – user24544 May 7 '18 at 12:43
  • $\begingroup$ I dont think my question is a duplicate - at least I did not find the answer to my question yet $\endgroup$ – user24544 May 7 '18 at 12:52
8
+50
$\begingroup$

For the first question, as Richard Hardy points out, there is an excellent blog post on the topic. There is also this post and this post which I have found very helpful.

For the second question, you need to take into account the two basic approaches to multistep times series forecasting: Recursive forecasting and direct forecasting:

  • In recursive forecasting (also called iterated forecasting) you train your model for one step ahead forecasts only. After the training is done you apply your final model recursively to forecast 1 step ahead, 2 steps ahead, etc...until you reach the desired $n$ steps forecast horizon. To do this, you feed the forecast from each successive step back into the model to generate the next step. This approach is used by traditional forecasting algorithms like ARIMA and Exponential Smoothing algorithms, and can be also used for Machine Learning based forecasting (see this post for an example, and this post for some discussion).
  • Direct forecasting is when you train a separate model for each step (so you trying to "directly" forecast the $n^{th}$ step ahead instead of reaching $n$ steps recursively. See Ben Taied et al. for a discussion of direct forecasting and more complex combined approaches.

Note that Hyndman's blog post on cross validation for time series covers both one step ahead and direct forecasting.


To clarify recursive forecasting (based on the comments):

  1. First you train your model.
  2. Once training is done, you take $[Y_1, Y_2,....Y_t]$ to calculate $\hat{Y}_{t+1}$ (this is your 1 step ahead forecast),
  3. then you use $[Y_2,..., Y_t,\hat{Y}_{t+1}]$ to calculate $\hat{Y}_{t+2}$, then $[Y_3,..., Y_t,\hat{Y}_{t+1}, \hat{Y}_{t+2}]$ to calculate $\hat{Y}_{t+3}$, and so on...until you reach $\hat{Y}_{t+n}$.

(Here $Y$ are actual values and $\hat{Y}$ are forecast values.)

$\endgroup$
  • $\begingroup$ Thanks, also for the links! Still some questions: (1) If I understand, the iterative forecast is lag n = 1, while the direct forecast is lag n > 1. Is this correct? Practically, when I am training/testing my model does this determine how much I have to shift in a supervised problem the time series (e.g. if I want to make a 5-step in ahead prediction, I use the timestamps t-5 to predict t)? (2) You mention using the predictions to iteratively update the model, I guess this must be the case both for lag n = 1, n > 5. However, once we apply the model I assume real incoming data can be used? $\endgroup$ – user24544 May 20 '18 at 8:03
  • $\begingroup$ @TestGuest "(1) If I understand, the iterative forecast is lag n = 1, while the direct forecast is lag n > 1. Is this correct? Practically, when I am training/testing my model does this determine how much I have to shift in a supervised problem the time series (e.g. if I want to make a 5-step in ahead prediction, I use the timestamps t-5 to predict t)?" - Yes correct. $\endgroup$ – Skander H. May 20 '18 at 18:46
  • $\begingroup$ @TestGuest "(2) You mention using the predictions to iteratively update the model, I guess this must be the case both for lag n = 1, n > 5. However, once we apply the model I assume real incoming data can be used? " -- **Not exactly. The model has been already trained in this case and no longer needs updating. You are applying an already trained model - you're simply feeding the outputs of the model back in recursively...[continued] $\endgroup$ – Skander H. May 20 '18 at 18:57
  • $\begingroup$ So first you $[Y_1, Y_2,....Y_t]$ to calculate $\hat{Y}_{t+1}$ (this is your 1 step ahead forecast), then you use $[Y_2,..., Y_t,\hat{Y}_{t+1}]$ to calculate $\hat{Y}_{t+2}$, then $[Y_3,..., Y_t,\hat{Y}_{t+1}, \hat{Y}_{t+2}]$ to calculate $\hat{Y}_{t+3}$, and so on until you reach $\hat{Y}_{t+n}$ (Here $Y$ are actual values and $\hat{Y}$ are forecast values. $\endgroup$ – Skander H. May 20 '18 at 18:57
  • $\begingroup$ Thank you! It seems however strange to me to not replace the predictions with real values, as soon as these are available -- if the goal is to have the best possible results. So, why is that? Thanks again.. $\endgroup$ – user24544 May 21 '18 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy