6
$\begingroup$

Question: Is it correct to use k-fold cross validation for choosing the best forecasting model among several regression models?

Case: Data come from a time series (e.g. prices), but I have access only to a random fraction of them. Instead of having 365data/year, I have only some of these observations in a casual order (e.g. 50 out of 365). Thus, I built several forecasting models (regression) based on external features that are available for every day avoiding a classical time series approach (ARIMA, etc.).

I have read that CV isn't adapt to time series due to autocorrelation issues; however, I'm wondering if this problem could affect my analysis.

What kind of test should I perform? What are potential problems I should pay attention to?

$\endgroup$
  • 2
    $\begingroup$ See robjhyndman.com/publications/cv-time-series and robjhyndman.com/hyndsight/tscv $\endgroup$ – Rob Hyndman Feb 1 '18 at 10:58
  • $\begingroup$ Thanks, Rob. My questions come up exactly after reading these two resources that I found very interesting. However, due to my lack of knowledge, I probably miss some points. Should I consider autocorrelation even if I use a simple regression model y = b0 + b1*x1 + b2*x2 where x1 and x2 do not represent any previous state of y (e.g. representing the price), but other features (e.g., temperature and wind)? $\endgroup$ – user193664 Feb 1 '18 at 12:59
6
$\begingroup$

Cross-validation is great! You can and should use cross-validation for this purpose. The trick is to perform cross-validation correctly for your data, and k-fold is too naive to deal with the autocorrelation.

You've correctly identified the fact that sequential data (like time series) will be subject to autocorrelation. In other words, the traditional supervised learning assumption of i.i.d. observations doesn't hold in this case.

In fact, most cross validation schemes appear to rely on having i.i.d. data because the training-test splits do not take time indices into account. For example, 5-fold cross validation applied naively over 5 time periods would ignore the sequential nature of time, mixing up past, present & future:

enter image description here

This would be wrong because

  • Your autoregressive models require a contiguous block of data, since they rely on the presence of autocorrelations at predefined lags (instead of having training sets split into 2 parts). Indeed, this is roughly the purpose of models like ARIMA - to capture autocorrelation in a way that many other models don't.
  • You should not train models on future data anyway, to avoid look-ahead bias

Hyndman (who has already commented on your question to post 2 great links), has lots of good examples of using a rolling or sliding window approach to cross validation to avoid this issue. For 5 time periods, you would split the sets as follows:

enter image description here

Another approach is to use an expanding window, though this may not be appropriate in your case:

enter image description here

Both of these schemes deal with the issues we identified earlier, and shouldn't be too hard to code up.

$\endgroup$
  • $\begingroup$ Thanks a lot, A.G.. Very useful explication. However, if I resort to the first of the two methods proposed, within the context of a model that involved both variables that change "little" but frequently in time (e.g. stock prices) and others that do not change over a long time period (once per year), it may be misleading. Indeed, to estimate the parameter associated to the variable presenting few changes along the time span available, I need at least to observed one change within the same time windows (Time 1, Time 2, etc.), which may be not guaranteed by this type of split. $\endgroup$ – user193664 Feb 1 '18 at 15:13
  • $\begingroup$ Indeed. In that case, I would recommend the second scheme (the expanding window) or trying to size the folds so that they incorporate a change in your long-running variable. $\endgroup$ – A. G. Feb 1 '18 at 15:39
  • $\begingroup$ Thank you for the graphic representation, it's really useful. What should I do next? I would like to get one single value for the loss function(e.g. prediction MSE). Would be correct to compute the prediction MSE for each split and then average them? $\endgroup$ – Juan Manuel Dec 20 '18 at 1:04
  • $\begingroup$ That’s right. MSE values can be averaged without any problems. $\endgroup$ – A. G. Dec 20 '18 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.