4
$\begingroup$

I have two time series, first is forecasted values (results of some forecasting algorithm) and second series is, actual values observed for same time frame.

We are trying to compare both these series and find out how much forecasted values are inline with actual values. We have used MAPE & MAE formulas which are fine when the difference between both are small values.

When difference between actual and forecast values are far off, MAE and MAPE are bigger values (I understand that MAPE has no upper limit, which is why bigger values).

These values make sense for statistician, but when common users saw these numbers, feedback we got was our accuracy calculation is doomed. Now the task we are trying to answer is, how can we calculate the difference between forecast vs actual (using MAE (or) MAPE (or) some other algorithm) and show it with in range of 0 - 100%?

Any suggestions would be appreciated.

EDIT 1: These numbers are handmade to convey the problem we are trying to solve, please don't consider about what forecasting algorithm may improve our forecast values etc., irrespective what best algorithm we use, there are few data sets we have could force us into this particular situation.

Here is example time series:

enter image description here

Improve this question
$\endgroup$
13
  • 1
    $\begingroup$ If we just look at the 9 observations you listed, your forecast is indeed poor, and a naive model would work better. But there may be a lot more to the series we're not seeing -- for example, these may be off-season forecasts, where the mean forecast and actual may both be, say, 2000. How do you subjectively feel about the quality of these forecasts? $\endgroup$
    – zbicyclist
    Commented Dec 23, 2014 at 22:23
  • 1
    $\begingroup$ Most forecasts are procedures that begin with fitting a model in a way that minimizes some measure of accuracy. You should prefer to use the same measure in assessing the quality of the forecast! $\endgroup$
    – whuber ♦
    Commented Dec 23, 2014 at 23:19
  • 1
    $\begingroup$ The best way is to compute the cost of forecast errors in units that business users understand. For instance, if it's sales forecast, then there could be a cost associated with error in dollars. The cost is usually asymmetric, btw. $\endgroup$
    – Aksakal
    Commented Dec 24, 2014 at 5:06
  • 1
    $\begingroup$ @Nambari, let's say actual is \$100, and your forecast was \$500. What is the accuracy in your understanding? $\endgroup$
    – Aksakal
    Commented Dec 24, 2014 at 16:01
  • 1
    $\begingroup$ @Nambari, it's not common to think in terms of accuracy where precise forecast is 100% and bad forecast is 0%. I think this is your issue with applying MAPE, you're not using it conventionally. $\endgroup$
    – Aksakal
    Commented Dec 24, 2014 at 16:12

1 Answer 1

Reset to default
5
$\begingroup$

One approach I've used for this problem is to define the MAPE as

(A-F)/(average of A and F)

instead of

(A-F)/A.

This measure (which I think I borrowed from Mosteller and Tukey's book, but I don't have it at hand right now) is symmetric and bounded by -200% and +200%. I know you wanted it to be 0 through 100, but I got you partway there with a measure I may be able to find a reference for.

I have used this where (a) I wanted a symmetric measure, and (b) where I wanted to cap the errors ['whether they were horrible (200%) or atrocious (5000%) didn't matter]. The image below compares a standard MAPE with this calculation (AdjMAPE). Later edit: because the errors are signed, they should be a form of MPE, not MAPE. See also comments below by me and whuber. enter image description here

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you for your time and answer, I think the approach you followed here is similar to sMAPE referred here en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error (but this seems not a real absolute percentage accuracy calculation based Rob J because results are in negative accuracy) otexts.org/fpp/2/5 $\endgroup$
    – kosa
    Commented Dec 23, 2014 at 23:06
  • 1
    $\begingroup$ Many analytical laboratories call this figure the "Relative Percent Difference": see (Google books) at books.google.com/… or books.google.com/…, for instance. $\endgroup$
    – whuber ♦
    Commented Dec 23, 2014 at 23:17
  • $\begingroup$ @whuber when you say this figure, you mean -200 to 200 (or) 0 to 100%? Will go through the links you posted, thanks for your time! $\endgroup$
    – kosa
    Commented Dec 23, 2014 at 23:18
  • 1
    $\begingroup$ It's precisely the formula given in this answer: the difference divided by the average. It's a measure that is appropriate when both quantities are subject to random error (of approximately the same magnitude). It tends to be near $0$ and in size much less than $100\%$, making it readily interpretable (that's one reason labs use it). Remember, it's measuring an error, so zero is best. RPDs of size greater than $100\%$ are usually considered pretty bad. $\endgroup$
    – whuber ♦
    Commented Dec 23, 2014 at 23:20
  • 2
    $\begingroup$ One would use absolute errors as an objective function for optimization, but signed errors to assess the associated residuals. Compare this with $\chi^2$ residuals in analyzing a two-way table, for instance: the $\chi^2$ statistic is a sum of non-negative terms $(O-E)^2/E$, but the residuals $(O-E)\sqrt{E}$ are given the signs of the differences $O-E$ so you can see which data are higher than expected and which are lower. $\endgroup$
    – whuber ♦
    Commented Dec 23, 2014 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.