1
$\begingroup$

Some measures of forecast accuracy, such as the mean absolute percentage error (MAPE), are "distorted" or are not defined, it the actual realization of the variable is close to zero, or equal to zero, respectively. This often happens with regard to growth rates, where for example GDP growth or Inflation is close to zero.

Moreover, some metrics, such as the RMSE, penalize outliers so much that for some purposes the metric becomes almost useless. Take for example the RMSE over 40 quarter rolling one year ahead forecasts. None of the major forecasters surveyed by Bloomberg anticipated the pandemic at the end of 2019 (for good reason), and once 2020 is included in the sample, the errors explode.

Are there any metrics in use that address either of these issues systematically, maybe even both? I am aware of the two symmetric mean absolute percentage errors (SMAPE1, SMAPE2) and SMASE. Are there others?

Note that I am not interested in model based forecast evaluation. Knowledge of how the forecasts are derived should not be required.

Improve this question
$\endgroup$
5
  • $\begingroup$ Hi BrsG I found this which might be somewhat helpful. otexts.com/fpp2/accuracy.html $\endgroup$
    – mlofton
    Aug 17, 2021 at 22:26
  • $\begingroup$ Thanks, that gives me the SMASE on top of what I already have. $\endgroup$
    – BrsG
    Aug 17, 2021 at 22:39
  • $\begingroup$ +1 but I wonder if it is a defect that the forecast models had huge errors once covid hit. $\endgroup$
    – Dave
    Aug 18, 2021 at 12:13
  • $\begingroup$ @Dave: there is indeed an incurable defect in the sense that these kind of events are unpredictable, or at least their timing is unpredictable. On top of that, the policy response (e.g. full lockdown) was largely unpredictable. In many case, it went over and beyond what the WTO had recommended for such scenarios. Forecasts improved the further we moved into 2020 because more information about policy measures and their effect was known. But all forecasters one-year ahead forecasts for 2020 were necessarily completely out of line. $\endgroup$
    – BrsG
    Aug 18, 2021 at 14:09
  • $\begingroup$ Having said that, Covid was just and example, I am looking for metrics that deal with that kind of situation in a systematic way, for example by weighing outliers less punitively, or by ignoring them altogether. Of course, it would be easy to "manually" ignore 2020, but there are cases that are much less obvious. $\endgroup$
    – BrsG
    Aug 18, 2021 at 14:13

1 Answer 1

Reset to default
1
$\begingroup$

Relative Total Absolute Error: mean of all absolute errors devided by the mean of the absolute actuals.

This measure is a MAPE where the sum is moved into the fraction. An advantage is that the RTAE can deal with zero actuals.

That said, SMAPE and the RTAE are useful when there are a few zeroes in your time series. If there a lot of zeroes; close to 50% or more, then these measures are not useful to evaluate the forecast. In such a case the best forecast would be zero according to these measures. Here is a reference for error measures when having time series with a lot of zeroes: https://www.lancaster.ac.uk/pg/waller/pdfs/Intermittent_Demand_Forecasting.pdf .

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.