8
$\begingroup$

Question: Is it correct to use k-fold cross validation for choosing the best forecasting model among several regression models?

Case: Data come from a time series (e.g. prices), but I have access only to a random fraction of them. Instead of having 365data/year, I have only some of these observations in a casual order (e.g. 50 out of 365). Thus, I built several forecasting models (regression) based on external features that are available for every day avoiding a classical time series approach (ARIMA, etc.).

I have read that CV isn't adapt to time series due to autocorrelation issues; however, I'm wondering if this problem could affect my analysis.

What kind of test should I perform? What are potential problems I should pay attention to?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ See robjhyndman.com/publications/cv-time-series and robjhyndman.com/hyndsight/tscv $\endgroup$
    – Rob Hyndman
    Feb 1, 2018 at 10:58
  • $\begingroup$ Thanks, Rob. My questions come up exactly after reading these two resources that I found very interesting. However, due to my lack of knowledge, I probably miss some points. Should I consider autocorrelation even if I use a simple regression model y = b0 + b1*x1 + b2*x2 where x1 and x2 do not represent any previous state of y (e.g. representing the price), but other features (e.g., temperature and wind)? $\endgroup$
    – user193664
    Feb 1, 2018 at 12:59
  • $\begingroup$ Potential duplicate of stats.stackexchange.com/q/14099/49798 $\endgroup$
    – william_grisaitis
    Dec 12, 2019 at 20:42
  • 1
    $\begingroup$ Does this answer your question? Using k-fold cross-validation for time-series model selection $\endgroup$
    – william_grisaitis
    Dec 12, 2019 at 20:42
  • $\begingroup$ @user193664 Re: "Should I consider autocorrelation" --- yes, because those covariates can be autocorrelated, and you are regressing y on them. For example, wind is probably autocorrelated (i.e. wind at time t is related to wind at time t-1, for a sufficiently small time difference). And if you linearly regress y on that covariate, your predictions will have that autocorrelation as well. $\endgroup$
    – william_grisaitis
    Dec 12, 2019 at 20:44

1 Answer 1

Reset to default
14
$\begingroup$

Cross-validation is great! You can and should use cross-validation for this purpose. The trick is to perform cross-validation correctly for your data, and k-fold is too naive to deal with the autocorrelation.

You've correctly identified the fact that sequential data (like time series) will be subject to autocorrelation. In other words, the traditional supervised learning assumption of i.i.d. observations doesn't hold in this case.

In fact, most cross validation schemes appear to rely on having i.i.d. data because the training-test splits do not take time indices into account. For example, 5-fold cross validation applied naively over 5 time periods would ignore the sequential nature of time, mixing up past, present & future:

enter image description here

This would be wrong because

  • Your autoregressive models require a contiguous block of data, since they rely on the presence of autocorrelations at predefined lags (instead of having training sets split into 2 parts). Indeed, this is roughly the purpose of models like ARIMA - to capture autocorrelation in a way that many other models don't.
  • You should not train models on future data anyway, to avoid look-ahead bias

Hyndman (who has already commented on your question to post 2 great links), has lots of good examples of using a rolling or sliding window approach to cross validation to avoid this issue. For 5 time periods, you would split the sets as follows:

enter image description here

Another approach is to use an expanding window, though this may not be appropriate in your case:

enter image description here

Both of these schemes deal with the issues we identified earlier, and shouldn't be too hard to code up.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks a lot, A.G.. Very useful explication. However, if I resort to the first of the two methods proposed, within the context of a model that involved both variables that change "little" but frequently in time (e.g. stock prices) and others that do not change over a long time period (once per year), it may be misleading. Indeed, to estimate the parameter associated to the variable presenting few changes along the time span available, I need at least to observed one change within the same time windows (Time 1, Time 2, etc.), which may be not guaranteed by this type of split. $\endgroup$
    – user193664
    Feb 1, 2018 at 15:13
  • 1
    $\begingroup$ Indeed. In that case, I would recommend the second scheme (the expanding window) or trying to size the folds so that they incorporate a change in your long-running variable. $\endgroup$
    – A. G.
    Feb 1, 2018 at 15:39
  • $\begingroup$ Thank you for the graphic representation, it's really useful. What should I do next? I would like to get one single value for the loss function(e.g. prediction MSE). Would be correct to compute the prediction MSE for each split and then average them? $\endgroup$
    – canovasjm
    Dec 20, 2018 at 1:04
  • $\begingroup$ That’s right. MSE values can be averaged without any problems. $\endgroup$
    – A. G.
    Dec 20, 2018 at 7:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.