Selecting the best (or more suitable to the user/client) output from a set of forecasts

I have approximately 3000 products for which I have to forecast in every, say, 2 months. I have the code in place for different forecasting models such as ARIMA, forced seasonal ARIMA, STLF etc.

Now for each product, I have forecasts coming out from 8 different models. Currently I use MAPE as a parameter to decide which forecast is the best. But sometimes I get a forecast which is a complete straight line and still has the least MAPE. These kinds of forecasts(straight line ones) don't really help me in making any decision. I also want the forecasts to capture seasonality which models like forced seasonal ARIMA do. But when I'm looking at the results of all the 3000 products, I do not have a perfect metric in place to see whether the output with best MAPE captures seasonality or not.

Is there a parameter which I can use, which can try to quantify the seasonality and MAPE together in the output and help me make a better decision of choosing the most appropriate model?

An example: I am forecasting for products (weekly, for 52 weeks) for, e.g., sales. I need to know how many will I sell each week. But if I get a flat line as my best output, I would not know exactly how much will I sell each week. This will stop me from estimating accurately how much space is required to keep these products. But if the 2nd best model, that has a little higher MAPE, but captures seasonality, then it enables me to decide that, okay, I need 'x' shelves to keep these products.

So these kind of small decisions, are more accurate if the seasonality is captured in the output. So I need to find a middle ground somewhere, a combination of some sort of MAPE and seasonality or any other metric which can help in deciding such things.

migrated from stackoverflow.comDec 17 '14 at 12:59

This question came from our site for professional and enthusiast programmers.

• What is your aim? One thing that you can do is to average the forecasts. Averaged forecasts give generally better predictions than individual ones. – Tim Dec 17 '14 at 13:02

Sometimes a flat line is the best forecast. Not everything is seasonal. Random variations can look like seasonality, but the standard tests, e.g., in R's auto.arima(), often do a pretty good job at deciding whether a given time series should be modeled using seasonality or not (however, see below on averaging). If you force seasonality, you may end up overfitting and getting bad forecasts.