Time series split (expanding window) vs k-fold cross validation in time series forecasting

Question

I do time series forecasting (q-o-q GDP) using ml models.

For hyper-parameter tunning I use grid search with cross-validation. Cross-validation is specifically Time series split (using expanding window approach). After getting model with best hyper-parameter combination I retrain the model once again on the whole train set and do prediction of one period ahead on one observation from test set, thus for testing the model I use expanding window as well. I have 2 questions:

1. is this correct approach for time series forecasting?
2. I also during hyper-parameter tunning used instead of time series split (expanding window) k-fold cross validation. I know that in theory this is wrong as model can train on future values and validate on past values, but when I test the final model on unseen test set using expanding window I get better results, than If I would for hyperparameter tunning use timeseries split (expanding window). So my question is, if this approach is still valid since I test the model correctly on unseen data while keeping time dependency.

Thank You.

Answer 1

Welcome to Cross Validated! :) To address your questions:

1. I would say that this approach for time series forecasting seems correct to me, but that also depends on how you're intending to use the model. In your application (forecasting quarter-over-quarter GDP growth(?)), your modeling/tuning/validation steps seem to be correct if your goal is to only predict the q-o-q GDP growth of the next quarter given some set of information (previous q-o-q GDP growths, etc.) since your validation methods line up with what your model would do in production, however, if your goal is to be able to......make multi-step ahead forecasts (the q-o-q GDP growth for the next next quarter, your model could perform well, but you also wouldn't have as much of an idea if it should perform well since you only validated on situations where the next value was predicted given past information, not the value of the next next quarter given the same set of information.

2. You are correct that conventional theory may have some difficulty with the use of regular k-fold cross validation in a time series context, in application, I feel like there can be some loosening of theory akin to "If it works (predicts well), it works (is correct enough to use regardless of theoretical violations." but unless some alternative methodology is carefully used (I'm not sure what this would be, but who am I to rule this out :) ), I feel like the use of future values to predict past values in time series training/validation (unless you're explicitly doing some form of backcasting, etc.) goes a bit over the line with regards to violation of theory even if predictive performance on unseen data is good/better than the alternative. This also goes against how I'm assuming your model will be used in practice since you seem to be using the past to predict the future and not the other way around (correct me if I'm wrong). You are correct in that you are respecting the time dependency by performing expanding window validation (I'm assuming in chronological order) on the test set, but I feel that by not respecting the time dependency during training, you lose some of the "guarantees" afforded to you by conventional theory, e.g., "A good test set score kindof-implies that my model will continue to perform well on future/to-be-recorded/etc. data". More concisely, even though your model that was trained without accounting for time dependency (trained on some predicting past given future, some predicting future given past) performed better (by whatever metric you were evaluating by) than the model that was trained to account for time dependency (only predicting future given past), I'm not sure if you have too much of a reason to believe that this discrepancy means anything for future performance since your data to make new future predictions will be received (I'm assuming) chronologically.

I hope that this answered your questions, feel free to ask for any clarification/point out any misunderstandings/incorrect assumptions and take care!