How do I interpret MRAE (Accuracy measure)?

Question

Can somebody explain to me how I would interpret the result of the MRAE.

In my textbook the MRAE is defined as followed:

$$ MRAE= {1 \over n}\left(\sum_{t=1}^n \left|{e_t \over e_t^*}\right|\right) $$

with $e_t=\text{actual value}-\text{forecasted value}$ and $e_t^*$ being the benchmarking method where $e_t^*=e_{t-1}$.

Now consider these simple example: Example 1: | actual | forecast | error | |--------|----------|---------| | 10 | 5 | 5 | |8 | 4 | 4 | |20 | 10| 10|

this would result in $$ MRAE= {1 \over 2}\cdot\left({4 \over 5}+{10 \over 4}\right)=1,65 $$

Example 2: | actual | forecast | error | |--------|----------|---------| | 10 | 1 | 9 | |8 | 1 | 7 | |20 | 1| 19|

this would result in $$ MRAE= {1 \over 2}\cdot\left({7 \over 9}+{19 \over 7}\right)=1,75 $$

Now what does this tell me? Am I 65% (75% respectively) better than the benchmarking method? And if so, what does that tell me about the actual forecasting method?

Thanks for your help.

Answer 1

The MRAE is a particularly unhelpful way to measure forecast accuracy. For any reasonable forecasting method, the errors can be positive or negative, so the distribution of the errors will have non-zero density at 0. Therefore the ratio of the two errors in the MRAE will have an undefined mean and infinite variance. Consequently the MRAE cannot converge and is not a consistent estimator of anything. This is one of the problems I pointed out in Hyndman & Koehler (2006).

I suggest you do not use the MRAE, despite some textbooks still including it.