I want to evaluate and compare how well various models perform with regards to modelling time series data (the data in question is daily revenue). It seems that cross validation error might be a reasonable way to do this.
I will quote this post, which nicely summarizes a procedure for time series cross validation:
An approach that's sometimes more principled for time series is forward chaining, where your procedure would be something like this:
- fold 1 : training , test 
- fold 2 : training [1 2], test 
- fold 3 : training [1 2 3], test 
- fold 4 : training [1 2 3 4], test 
- fold 5 : training [1 2 3 4 5], test 
My main question is:
For each of the various different models I am using, how do I establish, k, the minimum required size of the training set, in order for the evaluation of the model to be fair?
I sense that estimating k may cause problems; for example, one might set k such that the average error is reduced, when actually a larger error would be valid.
And a follow up question: is it valid to compare models that have vastly different k values?
E.g. for one model you might have an error vector of size 10, since k was so large, whereas for another you might have an error vector of size 1000.
Is it fair to compare the RMSE in this case?
I'd guess that it might be worth looking at the spread of the error?
The models I plan to compare are various different NN architectures, ARIMA stuff, ML techniques, along with some benchmark approaches, like mean, linear regression etc.
I don't have a vast amount of data, so often the NN models need to be trained on most of the data before they perform well (at least that is my intuition).
Any thoughts much appreciated!