I am trying to analyze a quite large (~25,000 rows) dataset of cash flow forecasts. Receipts and expenses are aggregated, thus I may end up with the following data:
Forecast = 6.0e+04
Actual = -5.0e+04
But also with
Forecast = 1.0e+06
Actual = 1.5e+06
or
Forecast = 1.0
Actual = 2e+06
As you can see, the actual can differ from the forecast in order of magnitude and even signs. However, I need to find a metric for the forecast error that works for all these cases and that is scale independent.
So far, I have used the Absolute Percentage Error, which works reasonably well for most of my data – but the few outliers (large forecast, small actual) render the mean absolute percentage error (MAPE) useless.
I then moved to the symmetric mean absolute percentage error:
mean(abs((act - forc)/(act + forc)))
This limits outliers as the output is between [0,1], but not if there's a sign change (=11 for the first example).
Are you aware of any metrics, that limit the influence of outliers while allowing to compare across forecasting horizons and series (scale independent) and that work with changing signs?