Selecting the best (or more suitable to the user/client) output from a set of forecasts I have approximately 3000 products for which I have to forecast in every, say, 2 months. I have the code in place for different forecasting models such as ARIMA, forced seasonal ARIMA, STLF etc.
Now for each product, I have forecasts coming out from 8 different models. Currently I use MAPE as a parameter to decide which forecast is the best. But sometimes I get a forecast which is a complete straight line and still has the least MAPE. These kinds of forecasts(straight line ones) don't really help me in making any decision. I also want the forecasts to capture seasonality which models like forced seasonal ARIMA do. But when I'm looking at the results of all the 3000 products, I do not have a perfect metric in place to see whether the output with best MAPE captures seasonality or not.
Is there a parameter which I can use, which can try to quantify the seasonality and MAPE together in the output and help me make a better decision of choosing the most appropriate model?
An example: I am forecasting for products (weekly, for 52 weeks) for, e.g., sales. I need to know how many will I sell each week. But if I get a flat line as my best output, I would not know exactly how much will I sell each week. This will stop me from estimating accurately how much space is required to keep these products. But if the 2nd best model, that has a little higher MAPE, but captures seasonality, then it enables me to decide that, okay, I need 'x' shelves to keep these products.
So these kind of small decisions, are more accurate if the seasonality is captured in the output. So I need to find a middle ground somewhere, a combination of some sort of MAPE and seasonality or any other metric which can help in deciding such things.
 A: Sometimes a flat line is the best forecast. Not everything is seasonal. Random variations can look like seasonality, but the standard tests, e.g., in R's auto.arima(), often do a pretty good job at deciding whether a given time series should be modeled using seasonality or not (however, see below on averaging). If you force seasonality, you may end up overfitting and getting bad forecasts.
To assess forecasting accuracy, use a holdout sample: keep your last 10 or 20 observations out of the training sample. Fit your eight models to the remaining observations, and forecast them out into the holdout sample. Check which model has the lowest Mean Absolute Deviation. Then refit this model on the whole sample and use it to forecast.
You may also want to look at taking all eight forecasts for a given time series and average them in each forecast time bucket. Averages of forecasts very often outperform selecting a "best" method. This is a common finding in the forecasting literature (see here and the references given there).
