4
$\begingroup$

I am testing a forecast framework which I have developed. I am using an ensemble model (mix of Linear, ETS, ARMA, Bayesian,) which was considerably better than mean forecasts when I was comparing them using Mean Absolute Percentage Error (MAPE) for a point by point difference over 500 different time series samples. However, I was asked to switch the error calculation to a cumulative sum at the end of forecast period (i.e. cumulative sum of forecast period vs cumulative sum of the actual values). And now it appears that the mean forecast is almost as good as the ensemble forecast! I am not sure if anyone else has also observed a similar behavior. What am I missing?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Sidhha, your problem as far as I can tell, arises largely out of using an ensemble model. In such a mixed model, MAPE from one of the underlying time series can affect the overall MAPE giving erroneous results. I would suggest that you try using a Symmetrical MAPE (sMAPE) or Weighted Absolute Percentage Error (WAPE). I am sure that the results will be consistent with the cumulative sums test that you are conducting.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I changed ensemble formulation to a single model and still see the same behavior. Do you still reckon it is due to MAPE? I was thinking on the lines that once you rollup the pointwise predictions to a cumulative sum, the effect of seasonality is averaged and so mean forecasts somehow seem to be as good as any other complicated model. Thoughts? $\endgroup$
    – Sidhha
    Commented May 20, 2015 at 1:31
  • $\begingroup$ Since it is scale dependent, cumulative sum cannot really provide a good picture as far as comparing forecast accuracy between two series is concerned. To compare two series I would suggest that you choose a scale independent metric. Let me know if this is clear. $\endgroup$
    – Raunak87
    Commented May 20, 2015 at 8:28
  • $\begingroup$ Totally agree to that. Wish I could explain it to my peers on the management side. :-/ $\endgroup$
    – Sidhha
    Commented May 20, 2015 at 8:43
  • 1
    $\begingroup$ Can't really help you there...if you can't convince them, confound them :) $\endgroup$
    – Raunak87
    Commented May 20, 2015 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.