2
$\begingroup$

I want to evaluate and compare how well various models perform with regards to modelling time series data (the data in question is daily revenue). It seems that cross validation error might be a reasonable way to do this.

I will quote this post, which nicely summarizes a procedure for time series cross validation:

An approach that's sometimes more principled for time series is forward chaining, where your procedure would be something like this:

  • fold 1 : training [1], test [2]
  • fold 2 : training [1 2], test [3]
  • fold 3 : training [1 2 3], test [4]
  • fold 4 : training [1 2 3 4], test [5]
  • fold 5 : training [1 2 3 4 5], test [6]

A more complete description of the procedure is outlined in this blog post, and this book.

My main question is:

For each of the various different models I am using, how do I establish, k, the minimum required size of the training set, in order for the evaluation of the model to be fair?

I sense that estimating k may cause problems; for example, one might set k such that the average error is reduced, when actually a larger error would be valid.

And a follow up question: is it valid to compare models that have vastly different k values?

E.g. for one model you might have an error vector of size 10, since k was so large, whereas for another you might have an error vector of size 1000.

Is it fair to compare the RMSE in this case?

I'd guess that it might be worth looking at the spread of the error?

The models I plan to compare are various different NN architectures, ARIMA stuff, ML techniques, along with some benchmark approaches, like mean, linear regression etc.

I don't have a vast amount of data, so often the NN models need to be trained on most of the data before they perform well (at least that is my intuition).

Any thoughts much appreciated!

$\endgroup$
0
$\begingroup$

I had the same question a while ago and landed on the last answer in the following link:

Divide a dataset into Training & Validation

I liked the theoretical basis for the idea of dividing the dataset into 63.2% training and 36.8% validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.