k-fold CV of forecasting financial time series -- is performance on last fold more relevant? I am working on an ANN-based forecasting model for a financial time series. I'm using 5-fold cross-validation and the average performance is so so. Performance on the last fold (the iteration where the last segment is omitted from training and used for validation) is better than average.
Is this a coincidence / data-dependent, or is validation performance on the last fold usually better? (presumably because training with all preceding data is more related to the subsequent data in time series)
This feels a bit like a weird question, but I'm hoping for some responses anyway. Thanks in advance :)
 A: In Sci-Kit Learn Python Kit they have something called "TimeSeriesSplit" which basically looks like the set of training/test samples you would get from a Walk Forward Optimization. Rob was right, you cannot use future datapoints to train for past test sets.... so the best way to cross-validate is to split your training sets into as many "folds" as possible while keeping the test set "Walking Forward". The consequence is have each successive training set a superset of those before it, and each test set more and more recent data to keep ahead of the "walk forward".
A: With time series, you cannot test a forecasting model via cross-validation in the normal way because you are then using future observations to predict the past. You must use only past observations to predict the future. The time series equivalent of LOO CV is to use a rolling forecast origin instead. I've written about it in this blog post. I'm not sure if k-fold CV has a direct time series equivalent.
A: I'm actually working on this same topic of the "Cross-validation with Financial Predictive Modeling" type of problem. So here are some of my findings.
Basically, I think that Cross-Validation by itself needs additional important considerations in order to produce valid and useful results, for the case of Financial Time Series Predictive Modeling.
A short answer for your question is in two parts:

*

*Performance on the last fold is better than average, sounds like an example of the Simpson's paradox


*The results you are getting could be due to back-test overfitting, and/or an information leakage between explanatory variables in the fold 2 and the target variable in the fold 1. Particularly in FTS, out-of-sample generalization does not guarantee out-of-distribution generalization.
More resources, reformulated questions, known effects in Financial Time Series.

*

*No unbiased estimator for the variance

We normally perform CV in order to increase our confidence in the model performance whenever new data arrives. Such endeavor ultimately leads to a trade-off we have to make (it would be good to do that consciously and explicitly), and that would be the Bias-Variance trade-off, which happens when we are trying to simultaneously minimize these Biased results and variance in the results (two different sources of error), being those the main reasons of preventing supervised learning algorithms from generalizing (learning) beyond your training data. Theoretically, we can have certainty that Cross-Validation does help to reduce bias, but we do not have (yet) a way to prove that there exists an estimator to express correctly some properties of the variance, so it goes frequently underestimated, as stated in this work. And so,

*

*The case of Financial Time Series (FTS)

In my current opinion, this work does provide various extra and special considerations when working with FTS. A quick list of important effects will be: Leakage of information, backtest-overfitting, Memory-loss. And the respective techniques, as proposed in the cited work, are: Purge&Embargo, Deflated performance metrics, fractional differentiation. I do mention those concepts because it could be the real causes of performance variation in your methods, a more "deep" reason since you are working with FTS, not just the cross-validation perspective by itself.

*

*Visual "classical" examples.
I am working on a python library that will provide methods, visualizations, and tests for this particular question of "What type of Cross-Validation is useful for FTS". Here are some early examples I draw for that.


Hope this comment helps new visitors, maybe not for answering the question directly (there can only be 1 accepted answer), but to provide more new questions, sources, and terms for further explorations.
