# Forecasting Prediction Accuracy

Out of 4 error paramters which one is best for evaluating prediction accuracy?

1. Average error
2. Mean absolute error
3. Mean squared error
4. Mean absolute % error

why?

• What's your criterion for "best"? – Andy May 17 '15 at 13:15
• All the 4 gives different value for the same set of training data. So which one should be used in what type of problem statement – Puneet Jindal May 17 '15 at 14:01
• Statistics isn't a "science of a single number" :) All of them tell you some information – petrbel May 17 '15 at 14:09

If you have only one time series, MAE (mean absolute error) is better than the alternatives because of the following (Davydenko and Fildes, 2016):

Fitting a statistical model usually delivers forecasts optimal under quadratic loss. This, e.g., happens when we fit a linear regression. If our density forecast from statistical modelling is symmetric, then forecasts optimal under quadratic loss are also optimal under linear loss. But, if we stabilise the variance by log-transformations and then transform back forecasts by exponentiation, we get forecasts optimal only under linear loss. If we use another loss, we must first obtain the density forecast using a statistical model, and then adjust our estimate given our specific loss function (see examples of doing this in Goodwin, 2000).

Let’s assume we want to empirically compare two methods and find out which method is better in terms of a symmetric linear loss (since this type of loss is commonly used in modelling). If we have only one time series, it seems natural to use a mean absolute error (MAE). Also, MAE is attractive as it is simple to understand and calculate (Hyndman, 2006).

If you want to evaluate accuracy across many series, the recommendation is (Davydenko and Fildes):

In order to overcome the disadvantages of existing measures, we recommend the use of the average relative MAE (AvgRelMAE) measure which is calculated as the geometric mean of relative MAE values.

A more detailed review can be found in (Davydenko and Fildes, 2014).

References

Davydenko, A., & Fildes, R. (2016).Forecast Error Measures: Critical Review and Practical Recommendations. In Business Forecasting: Practical Problems and Solutions.John Wiley & Sons Inc

The full text is available here.

Davydenko, A., & Fildes, R. (2014). Measuring Forecasting Accuracy: Problems and Recommendations (by the Example of SKU-Level Judgmental Adjustments). In Intelligent Fashion Forecasting Systems: Models and Applications(pp. 43-70). Springer Berlin Heidelberg.

The full text is available here.