Gaps of methods to evaluate prediction accuracy There are many methods to evaluate prediction models based in prediction errors, such as MSE, MAE, MAPE, WMAE, etc. These methods are usually used in data prediction competitions, where one is given a set of data used to discover potentially predictive relationships (training set), and must apply her predictions in another set of data used to assess the strength of her model (test data).
Some of these methods have gaps that can be explored. I can think of two cases:


*

*The Mean Absolute Error (MAE) does not take into account the relative size of the error, so I guess conservative predictions may not be a good idea if this is the evalutation measure.

*I am not sure how, but weighted measures such as WMAE may be explored since one may pay more attention in predicting values that are more important for the evaluation method.


Is my line of thought right? Can you elaborate more on that? What are other gaps that can be explored in the most common prediction evaluation measures and how one may explore this gaps? Is there a measure that you consider most appropriate for these data competitions?
 A: If you need to evaluate accuracy across many time series, you may find useful the following brief review:
Davydenko, A., & Fildes, R. (2016).Forecast Error Measures: Critical Review and Practical Recommendations. In Business Forecasting: Practical Problems and Solutions.John Wiley & Sons Inc.
The full text is available here.
A more detailed version of the above work was published in 2014:
Davydenko,  A.,  &  Fildes,  R.  (2014).  Measuring  Forecasting  Accuracy:  Problems  and  Recommendations  (by  the  Example  of  SKU-Level  Judgmental  Adjustments).  In Intelligent  Fashion Forecasting Systems: Models and Applications (pp. 43-70). Springer Berlin Heidelberg.
The full text is available here.
In brief, if you have many time series, the recommendation is (Davydenko and Fildes, 2016):

In order to overcome the disadvantages of existing measures, we recommend the use of the average relative MAE (AvgRelMAE) measure which is calculated as the geometric mean of relative MAE values.

If you have only one time series, (Davydenko and Fildes, 2016) recommend MAE :

Fitting a statistical model usually delivers forecasts optimal under quadratic loss. This, e.g., happens when we fit a linear regression. If our density forecast from statistical modelling is symmetric, then forecasts optimal under quadratic loss are also optimal under linear loss. But, if we stabilise the variance by log-transformations and then transform back forecasts by exponentiation, we get forecasts optimal only under linear loss. If we use another loss, we must first obtain the density forecast using a statistical model, and then adjust our estimate given our specific loss function (see examples of doing this in Goodwin, 2000).Let’s assume we want to empirically compare two methods and find out which method is better in terms of a symmetric linear loss (since this type of loss is commonly used in modelling). If we have only one time series, it seems natural to use a mean absolute error (MAE). Also, MAE is attractive as it is simple to understand and calculate (Hyndman, 2006).

Essentially, AvgRelMAE is just a way to extend the use of MAE to the case of evaluation across multiple time series.
However, the above publication notices that:

Potentially, MAE has the following limitation: absolute errors follow a highly skewed distribution with aheavy right tail, which means that MAE is not robust (in other words, it is a highly inefficient estimate).

(Davydenko and Fildes 2016) propose the following solution:

...MAE becomes a very inefficient estimate of the expected value of absolute error. One simple method to improve the efficiency of the estimates while not introducing substantial bias is to use asymmetric trimming algorithms, such as those described by (Alkhazeleh and Razali, 2010)

A: Something similar was my problem a while ago. I wrote this paper in this regard:
Probabilistic Forecasting with Stationary VAR Models. You might find it useful. You might also check Scoring Rule page in Wikipedia.
In short, probabilistic measure are theoretically more appropriate than MAE-like measures. They are generally a function of the forecast distribution (and not just the expected value of them, i.e., a point forecast). Of course, this is only applicable if the forecast scheme provides a probabilistic forecast.
The results of that paper shows that among different probabilistic rules, it seems that quadratic and logarithmic score rules are practically more stable or reliable. Of course, the final results are not sensitive to the choice of a probabilistic measure.
