2
$\begingroup$

We know that forward chaining a.k.a. time series cross validation is more appropriate than standard CV techniques in a time-series dataset.

However, there's relatively little discussion around the choice of inner CV loop of time series data when trying to evaluate the model's expected accuracy.

Generally speaking, what type of cross-validation is appropriate for the inner cross validation (e.g. hyperparam selection) that occurs? Should this also always be done in a forward chaining manner for best results?

Improve this question
$\endgroup$
10
  • $\begingroup$ What kind of distinction are you making between inner and outer cross validation? E.g. for selecting the tuning parameter of a LASSO model, there is only one round of validation needed, not two, or are there? $\endgroup$
    – Richard Hardy
    Commented Nov 12, 2016 at 9:36
  • $\begingroup$ @RichardHardy Yes, you could choose hyperparameters based on one round (e.g. best score on a holdout set), but I expect it'd be inferior to a CV approach for hyperparameter selection. Take sklearn's LassoCV, for instance. By default, 3-fold cross-validation is used, but any CV generator can be used that yields different train-test splits. This means you could be using yet another time-series CV approach here if you choose to. $\endgroup$
    – Brian Bien
    Commented Nov 12, 2016 at 16:59
  • $\begingroup$ No, I did not mean you split only once. E.g. for Leave-one-out CV (LOOCV) you split $n$ times where $n$ is your sample size; or for 3-fold CV you split 3 times. I count that as one round. Now that this is out of the way, how do you define your inner CV and your outer CV? $\endgroup$
    – Richard Hardy
    Commented Nov 12, 2016 at 17:35
  • $\begingroup$ Outer CV: let's say there are 3 splits using a time-series split- this implementation. Inside each of these splits, LassoCV is fit to the training data, where the inner CV is performed using the CV strategy provided to LassoCV. $\endgroup$
    – Brian Bien
    Commented Nov 12, 2016 at 17:50
  • 1
    $\begingroup$ So what are the goals of outer vs. inner CV, respectively? When there is only one round, the goal is to select a hyperparameter that maximizes the performance in the hold out samples. But how does this nested cross validation work? What is selected in inner vs. outer CV? After reading some more: is the inner CV used to select the parameters while outer CV to evaluate the performance "out of sample"? So train and test in inner, and then validate in outer? $\endgroup$
    – Richard Hardy
    Commented Nov 12, 2016 at 18:31

1 Answer 1

Reset to default
1
$\begingroup$

I do not quite see why the time series cross validation (TSCV) technique/design should depend on whether it is used for training the model or for evaluating its performance. But perhaps I am ignorant of something?

One rather simple and easy-to-use TSCV technique is the use of rolling windows. If we have a sample of $T$ observations, we may estimate the model using a window of $T_1<T$ consecutive observations and test or evaluate the model's performance by examining how well the model predicts the subsequent one or more observations for each window. So if you have a sample of 100, you could take

  • 1 though 70 as the first rolling window,
  • 2 though 71 as the second rolling windows,
  • ...,
  • 30 through 99 as the last rolling window,

and assess the predictive accuracy for observations 71, 72, ..., 100, respectively. This is just an example, the proportions of training and testing as well as the forecast horizons could be varied. Rob J. Hyndman provides an illustration in his blog post "Time series cross-validation: an R example".

However, there are alternatives. For example, the standard standard $K$-fold CV may be sensible even for time series data in certain setups. This is discussed in detail in Bergmeir et al. "A Note on the Validity of Cross-Validation for Evaluating Time Series Prediction" (working paper).

Improve this answer
$\endgroup$
4
  • $\begingroup$ This rolling window implementation is one of the candidates I was considering for the time series CV. The scikit-learn implementation is different in that it continues to reuse all prior training observations in consecutive iterations, resulting in more of a "growing" window than a sliding one. At any rate, the idea is the same. $\endgroup$
    – Brian Bien
    Commented Nov 13, 2016 at 15:50
  • $\begingroup$ That working paper is a great reference. It mentions that "in this way, the benefits of CV, especially for small datasets, cannot be exploited." The special case of inner CV is that you're training on a somewhat smaller sample whose time periods are somewhat closer together; hence my suspicion that traditional k-fold CV techniques may sometimes be more appropriate for inner CV. $\endgroup$
    – Brian Bien
    Commented Nov 13, 2016 at 15:53
  • $\begingroup$ @BrianBien, true the sample size issue becomes a concern. I find the working paper a bit unclear, but it is certainly interesting and the details can be worked out with some effort. $\endgroup$
    – Richard Hardy
    Commented Nov 13, 2016 at 16:46
  • $\begingroup$ @RichardHardy, I opened a question that encounter problem related to those addressed in this post. Can you give me your opinion about that? stats.stackexchange.com/questions/490240/… $\endgroup$
    – markowitz
    Commented Oct 7, 2020 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.