I am testing a forecast framework which I have developed. I am using an ensemble model (mix of Linear, ETS, ARMA, Bayesian,) which was considerably better than mean forecasts when I was comparing them using Mean Absolute Percentage Error (MAPE) for a point by point difference over 500 different time series samples. However, I was asked to switch the error calculation to a cumulative sum at the end of forecast period (i.e. cumulative sum of forecast period vs cumulative sum of the actual values). And now it appears that the mean forecast is almost as good as the ensemble forecast! I am not sure if anyone else has also observed a similar behavior. What am I missing?


Sidhha, your problem as far as I can tell, arises largely out of using an ensemble model. In such a mixed model, MAPE from one of the underlying time series can affect the overall MAPE giving erroneous results. I would suggest that you try using a Symmetrical MAPE (sMAPE) or Weighted Absolute Percentage Error (WAPE). I am sure that the results will be consistent with the cumulative sums test that you are conducting.

  • $\begingroup$ I changed ensemble formulation to a single model and still see the same behavior. Do you still reckon it is due to MAPE? I was thinking on the lines that once you rollup the pointwise predictions to a cumulative sum, the effect of seasonality is averaged and so mean forecasts somehow seem to be as good as any other complicated model. Thoughts? $\endgroup$ – Sidhha May 20 '15 at 1:31
  • $\begingroup$ Since it is scale dependent, cumulative sum cannot really provide a good picture as far as comparing forecast accuracy between two series is concerned. To compare two series I would suggest that you choose a scale independent metric. Let me know if this is clear. $\endgroup$ – Raunak87 May 20 '15 at 8:28
  • $\begingroup$ Totally agree to that. Wish I could explain it to my peers on the management side. :-/ $\endgroup$ – Sidhha May 20 '15 at 8:43
  • 1
    $\begingroup$ Can't really help you there...if you can't convince them, confound them :) $\endgroup$ – Raunak87 May 20 '15 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.