7
$\begingroup$

I'm doing product demand forecasting using 3 years' worth of daily sales data. I've built a few models and now I wish to test them. Based on Rob Hyndman's book and this resource, an expanding window walk forward cross validation is the gold standard for evaluating models in a time series context.

My question is, what's the most appropriate splitting? Should I train it initially on the first year (365 days) and do the walk forward test on the remaining 2 years (730 days)? What I expect will happen is that the error in forecasting will decrease as the training set grows, meaning the error will be large when the training set is small. Is it valid to train it initially on the first two years instead, and do the walk forward validation on the last year?

Thanks and any insight would be appreciated.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

Product demand data usually has a yearly seasonality. Training on the first year is not sufficient, as your model won't be able to capture any yearly seasonality or any long term trends. Most algorithms require at least 2 years of data for this reason (more would be better - but that's not always available for retail demand forecasting data).

At the same time you want to make sure that all of the seasonalities are present in your test set as well - so the optimal split in your case is 2 years training and 1 year testing.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have the same problem, but my data is not product demand data. It is predictive maintenance, so there is no seasonality. (Some items last less than 6 months.) What would be your recommendation for train/dev/test split for predicting failure in the next 30 days? $\endgroup$
    – arun
    Jul 5, 2018 at 14:51
  • 1
    $\begingroup$ @arun for predictive maintenance, a time series forecasting might not be the best approach. A survival analysis or reliability function calculation might provide a better approach. $\endgroup$
    – Skander H.
    Jul 5, 2018 at 17:00
0
$\begingroup$

Do you believe there is significant year to year variation?

If yes, it probably doesn't make sense to fit the model with only 1 year of data & a source of variation removed.

If no, which is generally unlikely due to seasonal trends, you might try it with 1 year of data. Using this small of a train set is a bit uncommon and may not be a reliable evaluation of parsimony.

In the end, if building & evaluating the model is computationally cheap, you can experiment with your idea and get the same validation results (plus an extra year) by starting at year one. If validation error is large in first year relative to the second, it suggests that either the model does not account for a large source of variation (perhaps your yearly effect), or that the model is overfit. If the validation error is similar between years one and two, it suggests that there may not be large yearly variation and the the model is not overfit.

Improve this answer
$\endgroup$
0
$\begingroup$

Daily data is frequently heavily dependent on daily habits i.e. it is more important to take into account deterministic structure while dealing with memory effect (same day last week for example ) . Daily data is also dependent on holiday effects (lead , contemporaneous and lags). In addition there are often monthly effects and level shift effects and trend effects. Often we have found that particular days of the month are important and even week-of-the month effects. We suggest a 3-4 year history to be able to tease out the regular while being robust to the irregular.

Forecast customer's spending is an interesting study that you might benefit from and here for more discussions https://stats.stackexchange.com/search?q=user%3A3382+daily+data.

In terms of splitting the data I wold probably initally use an 80/20 split and measure accuracies from many origins not just a sample of 1 origin to ensure a comprehensive/objective estimate of model performance as "one swallow does not a summer make "..

Improve this answer
$\endgroup$
5
  • $\begingroup$ I agree on many origins, but I think the OP will do a rolling-window approach (I take the walk-forward method to mean rolling windows). $\endgroup$
    – Richard Hardy
    May 1, 2018 at 13:59
  • $\begingroup$ @RichardHardy I'm currently doing an expanding window approach. Would you suggest doing a rolling window instead? $\endgroup$
    – meraxes
    May 1, 2018 at 14:05
  • $\begingroup$ @meraxes, not necessarily, but you should make it explicit in the question (for those who do not want to go over the linked material). $\endgroup$
    – Richard Hardy
    May 1, 2018 at 14:12
  • $\begingroup$ Wait, sorry @IrishStat, could you explain what you mean by "many origins"? $\endgroup$
    – meraxes
    May 17, 2018 at 11:49
  • $\begingroup$ If I had 100 observations and wanted to compute accuracies say for 5 period out horizon .. I would use 80 values to predict the next 5 ...81 values to predict the next 5 .... 95 values to predict the next 5. I would redevelop a model at each point thus the first model would be based upon 80 historical values , the second model would be based on 81 values ..the last model would be based upon 95 values. In this way I would have 16 estimates of a 5 period forecast error. The logic behind remodelling at each of the 16 origins is that the model/parameters and identified anomalies will/could change $\endgroup$
    – IrishStat
    May 17, 2018 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.