Can I roll-up forecasts to analyze against benchmark methods?

Question

There's a software product (call it Main) we use to allocate parts to various locations. The way it works is it sums up the demand history of all locations to the part level. For example, if part A has a historical parts distribution for Jan, Feb, Mar and Apr of (1,0,2,0) for location 1, and (3,1,0,0) for location 2, the software will sum up the demand to be (4,1,2,0) at the part level. From this it determines the best model from a list of 5, i.e., SES, DES, MA, WA and Croston, as the one with the smallest MSE. It then uses the chosen model to forecast down to the location level. So, if SES was chosen as the best model, then it will apply SES to the demand in location 1 and location 2 to build the forecast for May.

I realize that this isn't an ideal solution for forecasting parts and I have been asked to analyze the forecast accuracy of the method. I was going to roll-up the demand (as the software does) and use it to develop forecasts using a mean, naive and other commercial applications. My question is: Is it also all right to roll-up the forecasts that the Main software produces so that I have something to compare forecasts from my benchmark methods?

Answer 1

If I understand correctly, I believe you should not do this. Suppose you did something similar, with demand measured in 4 regions quarterly. Region A has 100 units in Q1, Region B 100 units in Q2, etc. and this is the case each year-- all orders are received in only one quarter for each region. If you aggregate, you will get a time series, and then a forecast, that appears to not have seasonality. From a production standpoint (if you produced everything in one place) that would be okay but from an inventory standpoint (assuming each region had its own warehoues) it would not be-- you should forecast them separately to pick up the seasonality of each.

In this case I think there could be a similar issue. If you need a separate forecast for each location then by aggregating you may get to a non-intermittent time series, leading you to use an exponential smoothing model, but then when you go back down to the location level you need an intermittent model like Croston's.

It would be better to use the appropriate model at both aggregation levels and then do a reconciliation across (like top-down or bottom-up) if you need to have forecasts that are equal at each level in the hierarchy.