3
$\begingroup$

I am trying to make a sales forecasting model to predict into the near future.

My data is monthly and of time series type as seen below:

  • 2013-1-1 : 2000$

  • 2013-2-1 : 6000$
    . . .

  • 2021-1-1 : 3000$

What I am thinking of trying first is a SARIMA model because my data have strong seasonality.

My concern is:

  • How should I split the dataset for training purposes?

The split should be chronological and I am thinking of doing a train/validation split and choosing the best model hyperparameters based on the validation RMSE. However, I have been advised to use a test set too for my model evaluation which is something I am cautious of.

Since I am purely choosing a model based on my validation data is a test set here really necessary? By doing this split I am using less data for training which I believe will result in a worse model.
What are your thoughts on this?

$\endgroup$

2 Answers 2

4
$\begingroup$

Eight years of data is a lot to work with. Using a test set may well be enlightening.

For instance, you could train various models on the first six years of data, then assess them using the seventh year. Then retrain them and forecast all models out into the eighth year. Then take a look at how well the holdout performance in the seventh year correlates with the holdout performance in the eighth year. It may well be that choosing the model that performed best in the seventh year did not lead to the model that would perform best in the eighth year.

Similarly, try this exercise earlier in your data, e.g., with a 5/1/1 or even a 4/1/1 split.

Also, consider seasonal exponential smoothing in addition to SARIMA. And think about averaging the forecasts from multiple models, which often improves accuracy.

$\endgroup$
1
$\begingroup$

Any time you use a dataset to fit some parameters, you are by definition fitting to that data.

  • Your training set fits each models parameters.
  • Your validation set fits which model has the best hyperparameters.

So, a test set will be necessary here as the model you choose will have been fit to the validation data. This will ensure that your model is performing well out-of-sample.

I would also second Stephan's answer that 8 years of monthly data is not too small so the gain from ensuring generalization will out-weigh the loss of validation data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.