I'm comparing some forecasting methods using four accuracy measures: Mean Absolute Error (MAE), Mean Squared Error (MSE), Mean Absolute Percentage Error (MAPE), Mean Absolute Scaled Error (MASE). The results are contradictory according to these different measures; which method finally should be selected? Is there any solution in this case?
Disagreement among these measures is actually a natural thing, as they target different objectives. Suppose you'd know the true probability distribution of the random variable (call it $Y$) of interest. Then, in order to minimize the MSE, you'd state the mean of $Y$ as a forecast. In order to minimize the MAE, however, you'd state the median of $Y$, which is different from the mean if the distribution of $Y$ is skewed.
Hence it is easily possible that method A gives better forecasts of the mean, whereas method B is better for the median, which makes the measures disagree. In order to choose an accuracy measure, you should think about which concept (mean vs median vs ...) you're interested in.
PS: MAPE and MASE seem to target more exotic objectives which are less popular than the mean and median. See http://arxiv.org/pdf/0912.0902.pdf for details on this.
It's quite common to have conflicting values for different error metrics like MAE, ME, RMSE, MAPE. There are two main reasons behind that
1) The way the formulas for those metrics are defined. 2) The nature of your data itself.
While you can't do much with regard to first reason.You should really examine the nature of your data, to decide on selecting the error metrics to compare different models.
Lets take a very basic example, suppose your data series has few values which are zero or very small ( close to zero), In that case MAPE( Mean Absolute Percentage Error) would not be a correct metric to compare your mode.Even though your forecast might be reasonably good, MAPE(both in sample /out of sample) could be very high due to those zero values.
Bottom Line : Examine your data.
The Problems I deal with : The previous approach works quite good if we have handful of time series. But what if we are dealing with selecting forecasting models for 300-500 time series ( a common case in supply chain, retail, manufacturing). Well the best approach is to calculate all the metrics for every model like ME, RMSE, MAPE, MPE, MAE, MASE( Mean Absolute Scaled Error) or any other metrics which is used in your domain.
Select the model which gives low values for most of the error metrics. This process can be automated.
For a list of modern error metrics you can refer [here](http://robjhyndman.com/papers/mase.pdf "