# Scale independent forecast error metric that works with changing signs

I am trying to analyze a quite large (~25,000 rows) dataset of cash flow forecasts. Receipts and expenses are aggregated, thus I may end up with the following data:

Forecast = 6.0e+04
Actual = -5.0e+04

But also with
Forecast = 1.0e+06
Actual = 1.5e+06

or
Forecast = 1.0
Actual = 2e+06

As you can see, the actual can differ from the forecast in order of magnitude and even signs. However, I need to find a metric for the forecast error that works for all these cases and that is scale independent.

So far, I have used the Absolute Percentage Error, which works reasonably well for most of my data – but the few outliers (large forecast, small actual) render the mean absolute percentage error (MAPE) useless. I then moved to the symmetric mean absolute percentage error:
mean(abs((act - forc)/(act + forc)))

This limits outliers as the output is between [0,1], but not if there's a sign change (=11 for the first example).

Are you aware of any metrics, that limit the influence of outliers while allowing to compare across forecasting horizons and series (scale independent) and that work with changing signs?

• have you tried to use the robustified z-scores? e.g. $(x_i-\mbox{med}(x))/\mbox{mad}(x)$ --where $x_i=$Forcast-Actual? – user603 Feb 11 '14 at 16:37
• Would you then go and eliminate outliers (|z| > 3) and continue using the MAPE? – hko Feb 12 '14 at 9:13
• yes. You can check the first part of this answer for more details. – user603 Feb 12 '14 at 9:37
• Thanks so far, that approach seems promising so far. Just quickly: is there any advantage to use the absolute error for $x_i$ or using $|x_i - med(x)|/mad(x)$ (how it is done in your linked answer)? – hko Feb 12 '14 at 11:02
• no, it's the same (you either flag outliers based on $|z_i|$ with $z_i=(x_i-\mbox{med}(x_i))/\mbox{mad}(x_i)$ or flag the outliers based on $z'_i$ with $z'_i=|x_i-\mbox{med}(x_i)|/\mbox{mad}(x_i)$ – user603 Feb 12 '14 at 11:15