I want to propose a simple experiment. Let's say I have a time series data, where I first split data into train and test sets and then work with my training set to pick the best model to do forecast analysis.
Usually we use cross-validation techniques to tune hyperparameters of a Machine Learning model, however, when dealing with complex time series structures, we can have a significant change in performance for a specific period of time and I would like to assess the stability that my model produces as it performance's varies over a specific window.
To clarify, let's say I use Rolling cross-validation, for h = 1, where: $ \hat{y}_{t+h} $ is my model's prediction and my metric is RMSE, so for 10 iterations, I would get 10 different RMSE for each timestamp: $ t_{1}, t_{2}, .., t_{10} $. And I generate a graph, like this:
The plot is not so important, it just servers to illustrate my question.
Now, my question is: Can I rely in a model where my RMSE curve is so unstable? Or should I trust on my initial train/test validation results and get only hyperparameters for this method?
Also, regarding feature importance, in Machine Learning: An Applied Econometric Approach the authors discuss how the coefficients choosed by a LASSO Regression model can vary a lot depending on CV split. Here I am trying to use this concept for time series as well, can I infer that features which are more stable through my whole cross-validation are in fact the best predictors for my target variable?
To summarize, when using big windows for cross-validation techniques, should I expect stability in a good performance model, or I only hope to get good results in a shot time window?
