# Selecting the best (or more suitable to the user/client) output from a set of forecasts

I have approximately 3000 products for which I have to forecast in every, say, 2 months. I have the code in place for different forecasting models such as ARIMA, forced seasonal ARIMA, STLF etc.

Now for each product, I have forecasts coming out from 8 different models. Currently I use MAPE as a parameter to decide which forecast is the best. But sometimes I get a forecast which is a complete straight line and still has the least MAPE. These kinds of forecasts(straight line ones) don't really help me in making any decision. I also want the forecasts to capture seasonality which models like forced seasonal ARIMA do. But when I'm looking at the results of all the 3000 products, I do not have a perfect metric in place to see whether the output with best MAPE captures seasonality or not.

Is there a parameter which I can use, which can try to quantify the seasonality and MAPE together in the output and help me make a better decision of choosing the most appropriate model?

An example: I am forecasting for products (weekly, for 52 weeks) for, e.g., sales. I need to know how many will I sell each week. But if I get a flat line as my best output, I would not know exactly how much will I sell each week. This will stop me from estimating accurately how much space is required to keep these products. But if the 2nd best model, that has a little higher MAPE, but captures seasonality, then it enables me to decide that, okay, I need 'x' shelves to keep these products.

So these kind of small decisions, are more accurate if the seasonality is captured in the output. So I need to find a middle ground somewhere, a combination of some sort of MAPE and seasonality or any other metric which can help in deciding such things.

## migrated from stackoverflow.comDec 17 '14 at 12:59

This question came from our site for professional and enthusiast programmers.

• What is your aim? One thing that you can do is to average the forecasts. Averaged forecasts give generally better predictions than individual ones. – Tim Dec 17 '14 at 13:02

Sometimes a flat line is the best forecast. Not everything is seasonal. Random variations can look like seasonality, but the standard tests, e.g., in R's auto.arima(), often do a pretty good job at deciding whether a given time series should be modeled using seasonality or not (however, see below on averaging). If you force seasonality, you may end up overfitting and getting bad forecasts.

To assess forecasting accuracy, use a holdout sample: keep your last 10 or 20 observations out of the training sample. Fit your eight models to the remaining observations, and forecast them out into the holdout sample. Check which model has the lowest Mean Absolute Deviation. Then refit this model on the whole sample and use it to forecast.

You may also want to look at taking all eight forecasts for a given time series and average them in each forecast time bucket. Averages of forecasts very often outperform selecting a "best" method. This is a common finding in the forecasting literature (see here and the references given there).

• Thank you for the answer. And I agree, sometimes a flat line is the best. But, a flat line may not help me in making small decisions which require a more accurate forecast probably. There are cases in which the second best(according to MAPE) captures seasonality as well. But because I have 3000 products, this thing will take a lot of time. That's why the need for a metric which can quantify the seasonality in the output as well. If you have thoughts on this, do let me know. – Anonymous Dec 17 '14 at 9:55
• Could you edit your question to give an example where a seasonal forecast is better for making decisions than a better nonseasonal forecast? I have to admit that I don't understand that. – Stephan Kolassa Dec 17 '14 at 9:57
• Sorry if it wasn't clear. I've edited the question now. – Anonymous Dec 17 '14 at 10:06
• Thanks. I still don't understand. Why does a flat line not allow you to estimate shelf space requirements? (They will simply not change over time.) Why is a seasonal forecast better for this although it is a worse forecast? It may be worthwhile to think some more about why exactly you want seasonality. (Incidentally, I have been forecasting retail sales for almost a decade now. Often, clients don't like a flat line because it doesn't look "sophisticated" enough - "why did I pay a lot of money to get a flat line?" - although it is the best forecast and gives the best inventory position.) – Stephan Kolassa Dec 17 '14 at 10:25
• Won't the shelf space requirements change? I think now it comes down to the accuracy of the space required. I believe that the space required can be estimated more accurately if I have seasonality in output. suppose I have the output as 2,2,2,2 for 4 weeks as the best output. And then I have the output as 1,2,5,0. For the 4th week in this case, I don't need any shelves for this product. Your thoughts? – Anonymous Dec 17 '14 at 10:31