What is the best way to get a measure of how well an ARIMA model can predict a timeseries when doing an out-of-sample rolling window? I cant use MPE cause it contains zeroes. What I am looking for is actually a span that states: " The prediction has an average error span between ... % and ... %. How do one find that out?
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
Symmetric mape (SMAPE) will be useful for you . Pursue https://stats.stackexchange.com/search?q=SMAPE .
-
1$\begingroup$ Caution with the SMAPE: if an actual is 0, then the SMAPE for that actual is 200%, indifferent of the forecast: (A=0:) |F-A|/(|A|+|F|) = |F|/|F| = 1. So, the SMAPE tolerates zeros in the actuals, but does not help to differentiate good and bad forecasts if the actuals are zero, since the SMAPE is the same regardless of the forecast in such a case. $\endgroup$– RubenDec 6, 2019 at 15:47
$\begingroup$
$\endgroup$
@IrishStat's suggestion to use SMAPE is probably the most correct.
A log(x+1) transform could also work.
It would be useful to know your application. What is the proportion of 0's? I am curious if you have an intermittent time series.
Intermittent time series are often modeled with Crostons method or the Poisson distribution, the benchmark forecast being an all-zero forecast. If you are dealing with intermittent time series then it is common to use a distributional error metric. Continuous rank probability score or pinball loss.
$\begingroup$
$\endgroup$
Another interesting measure that can deal with zero actuals:
Relative Total Absolute Error: mean of all absolute errors devided by the mean of the absolute actuals.
This measure is a MAPE where the sum is moved into the fraction. An advantage is that the RTAE can deal with zero actuals.
That said, SMAPE and the RTAE are useful when there are a few zeroes in your time series. If there a lot of zeroes; close to 50% or more, then these measures are not useful to evaluate the forecast. In such a case the best forecast would be zero according to these measures. Here is a reference for error measures when having time series with a lot of zeroes: https://www.lancaster.ac.uk/pg/waller/pdfs/Intermittent_Demand_Forecasting.pdf .
