My team is currently using a demand forecasting system that uses various exponential smoothing models and produces a forecast for each product and location (so a total number of times series running in the millions).
We have tested some other models such as ARIMA, NNets, GAM, etc...on individual times series, and for some product location combinations they seem to work better (in terms of accuracy measured using out of sample MAPE, RMSE and MAE) than our current system, for others they are comparable or worse than our current system.
Short of running each of the new models for each and every product/location combination and then calculating an aggregate RMSE, MAPE and MAE, how do we go about evaluating whether one of the new methods will work better at scale than our current system?