2
$\begingroup$

I am trying to analyze a quite large (~25,000 rows) dataset of cash flow forecasts. Receipts and expenses are aggregated, thus I may end up with the following data:

Forecast = 6.0e+04
Actual = -5.0e+04

But also with
Forecast = 1.0e+06
Actual = 1.5e+06

or
Forecast = 1.0
Actual = 2e+06

As you can see, the actual can differ from the forecast in order of magnitude and even signs. However, I need to find a metric for the forecast error that works for all these cases and that is scale independent.

So far, I have used the Absolute Percentage Error, which works reasonably well for most of my data – but the few outliers (large forecast, small actual) render the mean absolute percentage error (MAPE) useless. I then moved to the symmetric mean absolute percentage error:
mean(abs((act - forc)/(act + forc)))

This limits outliers as the output is between [0,1], but not if there's a sign change (=11 for the first example).

Are you aware of any metrics, that limit the influence of outliers while allowing to compare across forecasting horizons and series (scale independent) and that work with changing signs?

$\endgroup$
7
  • $\begingroup$ have you tried to use the robustified z-scores? e.g. $(x_i-\mbox{med}(x))/\mbox{mad}(x)$ --where $x_i=$Forcast-Actual? $\endgroup$
    – user603
    Feb 11, 2014 at 16:37
  • $\begingroup$ Would you then go and eliminate outliers (|z| > 3) and continue using the MAPE? $\endgroup$
    – hko
    Feb 12, 2014 at 9:13
  • $\begingroup$ yes. You can check the first part of this answer for more details. $\endgroup$
    – user603
    Feb 12, 2014 at 9:37
  • $\begingroup$ Thanks so far, that approach seems promising so far. Just quickly: is there any advantage to use the absolute error for $x_i$ or using $|x_i - med(x)|/mad(x)$ (how it is done in your linked answer)? $\endgroup$
    – hko
    Feb 12, 2014 at 11:02
  • $\begingroup$ no, it's the same (you either flag outliers based on $|z_i|$ with $z_i=(x_i-\mbox{med}(x_i))/\mbox{mad}(x_i)$ or flag the outliers based on $z'_i$ with $z'_i=|x_i-\mbox{med}(x_i)|/\mbox{mad}(x_i)$ $\endgroup$
    – user603
    Feb 12, 2014 at 11:15

1 Answer 1

1
$\begingroup$

You could look at the Mean Absolute Scaled Error (MASE - see its tag wiki for more information). Basically, it scales the Mean Absolute Error (MAE) by the MAE that the naive random walk would have achieved in-sample.

To deal with your outliers, you could either use the median instead of the mean, or use a trimmed or winsorized mean.

However, I'd rather try to understand where an actual of 2e6 with a forecast of 1 came from. This sounds like it could have major implications on whatever process relies on your forecast, whether or not you include it in your accuracy calculation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.