I'm comparing some forecasting methods using four accuracy measures: Mean Absolute Error (MAE), Mean Squared Error (MSE), Mean Absolute Percentage Error (MAPE), Mean Absolute Scaled Error (MASE). The results are contradictory according to these different measures; which method finally should be selected? Is there any solution in this case?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Do you have any measure/estimate of the consequences of being out by a given amount in your forecast? $\endgroup$– Scortchi ♦Commented Jan 19, 2014 at 20:35
-
$\begingroup$ NO, I don't have any measure for that end, Just I do the forecast using some methods (like SES, Theta, Naive,..) and I compare their accuracy using the mentioned measures $\endgroup$– RojiCommented Jan 20, 2014 at 13:01
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Disagreement among these measures is actually a natural thing, as they target different objectives. Suppose you'd know the true probability distribution of the random variable (call it $Y$) of interest. Then, in order to minimize the MSE, you'd state the mean of $Y$ as a forecast. In order to minimize the MAE, however, you'd state the median of $Y$, which is different from the mean if the distribution of $Y$ is skewed.
Hence it is easily possible that method A gives better forecasts of the mean, whereas method B is better for the median, which makes the measures disagree. In order to choose an accuracy measure, you should think about which concept (mean vs median vs ...) you're interested in.
PS: MAPE and MASE seem to target more exotic objectives which are less popular than the mean and median. See http://arxiv.org/pdf/0912.0902.pdf for details on this.
-
$\begingroup$ +1. Note that the expected absolute percentage error (APE) is minimized by the (-1)-median of the future distribution, see p. 752 in the published version of Gneiting's paper. The MASE actually doesn't target anything exotic at all. It's a scaled version of the MAE - scaled by the in-sample error of the random walk forecast, which is a scaling factor that is independent of the forecast and the future distribution. Thus, the expected MASE is minimized by the same quantity as the expected MAE, namely the median. $\endgroup$ Commented Feb 9, 2016 at 9:26
$\begingroup$
$\endgroup$
It's quite common to have conflicting values for different error metrics like MAE, ME, RMSE, MAPE. There are two main reasons behind that
1) The way the formulas for those metrics are defined. 2) The nature of your data itself.
While you can't do much with regard to first reason.You should really examine the nature of your data, to decide on selecting the error metrics to compare different models.
Lets take a very basic example, suppose your data series has few values which are zero or very small ( close to zero), In that case MAPE( Mean Absolute Percentage Error) would not be a correct metric to compare your mode.Even though your forecast might be reasonably good, MAPE(both in sample /out of sample) could be very high due to those zero values.
Bottom Line : Examine your data.
The Problems I deal with : The previous approach works quite good if we have handful of time series. But what if we are dealing with selecting forecasting models for 300-500 time series ( a common case in supply chain, retail, manufacturing). Well the best approach is to calculate all the metrics for every model like ME, RMSE, MAPE, MPE, MAE, MASE( Mean Absolute Scaled Error) or any other metrics which is used in your domain.
Select the model which gives low values for most of the error metrics. This process can be automated.
For a list of modern error metrics you can refer [here](http://robjhyndman.com/papers/mase.pdf "
