3
$\begingroup$

My team is currently using a demand forecasting system that uses various exponential smoothing models and produces a forecast for each product and location (so a total number of times series running in the millions).

We have tested some other models such as ARIMA, NNets, GAM, etc...on individual times series, and for some product location combinations they seem to work better (in terms of accuracy measured using out of sample MAPE, RMSE and MAE) than our current system, for others they are comparable or worse than our current system.

Short of running each of the new models for each and every product/location combination and then calculating an aggregate RMSE, MAPE and MAE, how do we go about evaluating whether one of the new methods will work better at scale than our current system?

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you think about my answer? $\endgroup$
    – Richard Hardy
    Jun 1, 2018 at 20:02
  • $\begingroup$ @RichardHardy I haven't marked it as accepted because I was hoping somebody would respond (nothing wrong with your, just the childish wishful thinking part of me thinking "there must be an easier way"). $\endgroup$
    – Skander H.
    Jun 1, 2018 at 21:18

1 Answer 1

Reset to default
3
$\begingroup$

If running all the models on all the time series is too expensive, you could pick a representative sample of time series and evaluate the models on each of the time series in that sample, hoping that the results will be representative for the whole population of time series.

But how do we pick that sample?

If you have an overview of the types of series and their counts, you could do random stratified sampling, i.e. randomly pick some series from each type and use weights to account for overrepresentation or underrepresentation if needed. If you do not have an overview, then select the series randomly (number all of the series and draw a bunch of random numbers between 1 and the number of series).

Improve this answer
$\endgroup$
3
  • $\begingroup$ But how do we pick that sample? $\endgroup$
    – Skander H.
    May 2, 2018 at 17:28
  • $\begingroup$ How big should the sample be so that we can decide on the model with confidence? $\endgroup$
    – Skander H.
    Jun 1, 2018 at 21:25
  • $\begingroup$ @Alex, you can see how much the model performance varies, e.g. estimate the variance of the performance measure, and then increase the sample size until you reach a value that is low enough for you. $\endgroup$
    – Richard Hardy
    Jun 5, 2018 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.