I am testing a forecast framework which I have developed. I am using an ensemble model (mix of Linear, ETS, ARMA, Bayesian,) which was considerably better than mean forecasts when I was comparing them using Mean Absolute Percentage Error (MAPE) for a point by point difference over 500 different time series samples. However, I was asked to switch the error calculation to a cumulative sum at the end of forecast period (i.e. cumulative sum of forecast period vs cumulative sum of the actual values). And now it appears that the mean forecast is almost as good as the ensemble forecast! I am not sure if anyone else has also observed a similar behavior. What am I missing?
Sidhha, your problem as far as I can tell, arises largely out of using an ensemble model. In such a mixed model, MAPE from one of the underlying time series can affect the overall MAPE giving erroneous results. I would suggest that you try using a Symmetrical MAPE (sMAPE) or Weighted Absolute Percentage Error (WAPE). I am sure that the results will be consistent with the cumulative sums test that you are conducting.