2
$\begingroup$

I am trying to analyze a quite large (~25,000 rows) dataset of cash flow forecasts. Receipts and expenses are aggregated, thus I may end up with the following data:

Forecast = 6.0e+04
Actual = -5.0e+04

But also with
Forecast = 1.0e+06
Actual = 1.5e+06

or
Forecast = 1.0
Actual = 2e+06

As you can see, the actual can differ from the forecast in order of magnitude and even signs. However, I need to find a metric for the forecast error that works for all these cases and that is scale independent.

So far, I have used the Absolute Percentage Error, which works reasonably well for most of my data – but the few outliers (large forecast, small actual) render the mean absolute percentage error (MAPE) useless. I then moved to the symmetric mean absolute percentage error:
mean(abs((act - forc)/(act + forc)))

This limits outliers as the output is between [0,1], but not if there's a sign change (=11 for the first example).

Are you aware of any metrics, that limit the influence of outliers while allowing to compare across forecasting horizons and series (scale independent) and that work with changing signs?

Improve this question
$\endgroup$
7
  • $\begingroup$ have you tried to use the robustified z-scores? e.g. $(x_i-\mbox{med}(x))/\mbox{mad}(x)$ --where $x_i=$Forcast-Actual? $\endgroup$ – user603 Feb 11 '14 at 16:37
  • $\begingroup$ Would you then go and eliminate outliers (|z| > 3) and continue using the MAPE? $\endgroup$ – hko Feb 12 '14 at 9:13
  • $\begingroup$ yes. You can check the first part of this answer for more details. $\endgroup$ – user603 Feb 12 '14 at 9:37
  • $\begingroup$ Thanks so far, that approach seems promising so far. Just quickly: is there any advantage to use the absolute error for $x_i$ or using $|x_i - med(x)|/mad(x)$ (how it is done in your linked answer)? $\endgroup$ – hko Feb 12 '14 at 11:02
  • $\begingroup$ no, it's the same (you either flag outliers based on $|z_i|$ with $z_i=(x_i-\mbox{med}(x_i))/\mbox{mad}(x_i)$ or flag the outliers based on $z'_i$ with $z'_i=|x_i-\mbox{med}(x_i)|/\mbox{mad}(x_i)$ $\endgroup$ – user603 Feb 12 '14 at 11:15
1
$\begingroup$

You could look at the Mean Absolute Scaled Error (MASE - see its tag wiki for more information). Basically, it scales the Mean Absolute Error (MAE) by the MAE that the naive random walk would have achieved in-sample.

To deal with your outliers, you could either use the median instead of the mean, or use a trimmed or winsorized mean.

However, I'd rather try to understand where an actual of 2e6 with a forecast of 1 came from. This sounds like it could have major implications on whatever process relies on your forecast, whether or not you include it in your accuracy calculation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.