1
$\begingroup$

We know that forward chaining a.k.a. time series cross validation is more appropriate than standard CV techniques in a time-series dataset.

However, there's relatively little discussion around the choice of inner CV loop of time series data when trying to evaluate the model's expected accuracy.

Generally speaking, what type of cross-validation is appropriate for the inner cross validation (e.g. hyperparam selection) that occurs? Should this also always be done in a forward chaining manner for best results?

$\endgroup$
10
  • $\begingroup$ What kind of distinction are you making between inner and outer cross validation? E.g. for selecting the tuning parameter of a LASSO model, there is only one round of validation needed, not two, or are there? $\endgroup$ – Richard Hardy Nov 12 '16 at 9:36
  • $\begingroup$ @RichardHardy Yes, you could choose hyperparameters based on one round (e.g. best score on a holdout set), but I expect it'd be inferior to a CV approach for hyperparameter selection. Take sklearn's LassoCV, for instance. By default, 3-fold cross-validation is used, but any CV generator can be used that yields different train-test splits. This means you could be using yet another time-series CV approach here if you choose to. $\endgroup$ – Brian Bien Nov 12 '16 at 16:59
  • $\begingroup$ No, I did not mean you split only once. E.g. for Leave-one-out CV (LOOCV) you split $n$ times where $n$ is your sample size; or for 3-fold CV you split 3 times. I count that as one round. Now that this is out of the way, how do you define your inner CV and your outer CV? $\endgroup$ – Richard Hardy Nov 12 '16 at 17:35
  • $\begingroup$ Outer CV: let's say there are 3 splits using a time-series split- this implementation. Inside each of these splits, LassoCV is fit to the training data, where the inner CV is performed using the CV strategy provided to LassoCV. $\endgroup$ – Brian Bien Nov 12 '16 at 17:50
  • 1
    $\begingroup$ So what are the goals of outer vs. inner CV, respectively? When there is only one round, the goal is to select a hyperparameter that maximizes the performance in the hold out samples. But how does this nested cross validation work? What is selected in inner vs. outer CV? After reading some more: is the inner CV used to select the parameters while outer CV to evaluate the performance "out of sample"? So train and test in inner, and then validate in outer? $\endgroup$ – Richard Hardy Nov 12 '16 at 18:31
1
$\begingroup$

I do not quite see why the time series cross validation (TSCV) technique/design should depend on whether it is used for training the model or for evaluating its performance. But perhaps I am ignorant of something?

One rather simple and easy-to-use TSCV technique is the use of rolling windows. If we have a sample of $T$ observations, we may estimate the model using a window of $T_1<T$ consecutive observations and test or evaluate the model's performance by examining how well the model predicts the subsequent one or more observations for each window. So if you have a sample of 100, you could take

  • 1 though 70 as the first rolling window,
  • 2 though 71 as the second rolling windows,
  • ...,
  • 30 through 99 as the last rolling window,

and assess the predictive accuracy for observations 71, 72, ..., 100, respectively. This is just an example, the proportions of training and testing as well as the forecast horizons could be varied. Rob J. Hyndman provides an illustration in his blog post "Time series cross-validation: an R example".

However, there are alternatives. For example, the standard standard $K$-fold CV may be sensible even for time series data in certain setups. This is discussed in detail in Bergmeir et al. "A Note on the Validity of Cross-Validation for Evaluating Time Series Prediction" (working paper).

$\endgroup$
4
  • $\begingroup$ This rolling window implementation is one of the candidates I was considering for the time series CV. The scikit-learn implementation is different in that it continues to reuse all prior training observations in consecutive iterations, resulting in more of a "growing" window than a sliding one. At any rate, the idea is the same. $\endgroup$ – Brian Bien Nov 13 '16 at 15:50
  • $\begingroup$ That working paper is a great reference. It mentions that "in this way, the benefits of CV, especially for small datasets, cannot be exploited." The special case of inner CV is that you're training on a somewhat smaller sample whose time periods are somewhat closer together; hence my suspicion that traditional k-fold CV techniques may sometimes be more appropriate for inner CV. $\endgroup$ – Brian Bien Nov 13 '16 at 15:53
  • $\begingroup$ @BrianBien, true the sample size issue becomes a concern. I find the working paper a bit unclear, but it is certainly interesting and the details can be worked out with some effort. $\endgroup$ – Richard Hardy Nov 13 '16 at 16:46
  • $\begingroup$ @RichardHardy, I opened a question that encounter problem related to those addressed in this post. Can you give me your opinion about that? stats.stackexchange.com/questions/490240/… $\endgroup$ – markowitz Oct 7 '20 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.