Correctly evaluating (different) time series models

Question

Currently, I am going through Hyndman's book to learn more about time series (I use the 2nd version with the forecast package). I have trouble understanding the "correct" way of evaluating multiple (let's say 5 different) time series models. I can think of 3 different ways, and I am not sure which is the correct one:

1. Split data into train/test => Fit all models on the training set and evaluate the models on the test set.

Even though this would give us an indication of which model is the best, it will estimate the test error too small since we made a model choice based on it.

2. Use time series cross validation as described in that part.

This helps us finding the best model based on the lowest CV error. But after retraining the "best" model on the whole training data we don't have a test set left.

3. Split the data into train/test => Fit all models on train and use time series CV on train => After finding the best model, retrain on the best model on train data set and evaluate it on the test data.

In my opinion, the last approach would be the correct one if our goal is to find the best model and also get a reasonable error on a test set. Is my intuition correct, or am I missing something here? I don't know if I think too complicated?

I have a second question that is closely related to this one: Is it possible to use the tsCV function from forecast on a combination of forecast models/hybrid model (e.g. the average forecast of an ARIMA, ETS, other models). In other words, the same as in this chapter but not evaluated on a test set but with tsCV.

Answer 1

As far as I know the problem of train/validate/test has received more attention in the "classic" ML realm than in the time series realm. Some thoughts:

You almost always need to do "time series cross validation" to get reliable error estimates for time series. Problem: most of the time you don't have enough data to do a proper backtest for validation & test on different data slices. You will easily backtest overfit without realising it. Marco Lopez de Prado published good stuff about the topic (but mostly specific to the finance domain).

Your third option will choose a good model, but the error estimate on the probably small test most likely will be unrealistic.

Depending on your data and models you might get better results the other way around. Hypertuning parameters often can be done with relatively few data from a small validation set and then you can estimate the resulting error with a full backtest.

From my experience you definitely don't think too complicated. These issues are very real. If a correct validation scheme exists, it is probably impractical. You have to make trade-offs and these depend on your data, models and your tuning needs.