Why use a certain measure of forecast error (e.g. MAD) as opposed to another (e.g. MSE)?

Question

MAD = Mean Absolute Deviation MSE = Mean Squared Error

I've seen suggestions from various places that MSE is used despite some undesirable qualities (e.g. http://www.stat.nus.edu.sg/~staxyc/T12.pdf, which states on p8 "It is commonly believed that MAD is a better criterion than MSE. However, mathematically MSE is more convenient than MAD.")

Is there more to it than that? Is there a paper that thoroughly analyzes the situations in which various methods of measuring forecast error are more/less appropriate? My google searches haven't revealed anything.

A similar question to this was asked at https://stackoverflow.com/questions/13391376/how-to-decide-the-forecasting-method-from-the-me-mad-mse-sde, and the user was asked to post on stats.stackexchange.com, but I don't think they ever did.

Answer 1

To decide which point forecast error measure to use, we need to take a step back. Note that we don't know the future outcome perfectly, nor will we ever. So the future outcome follows a probability distribution. Some forecasting methods explicitly output such a full distribution, and some don't - but it is always there, if only implicitly.

Now, we want to have a good error measure for a point forecast. Such a point forecast $F_t$ is our attempt to summarize what we know about the future distribution (i.e., the predictive distribution) at time $t$ using a single number, a so-called functional of the future density. The error measure then is a way to assess the quality of this single number summary.

So you should choose an error measure that rewards "good" one number summaries of (unknown, possibly forecasted, but possibly only implicit) future densities.

The challenge is that different error measures are minimized by different functionals. The expected MSE is minimized by the expected value of the future distribution. The expected MAD is minimized by the median of the future distribution. Thus, if you calibrate your forecasts to minimize the MAE, your point forecast will be the future median, not the future expected value, and your forecasts will be biased if your future distribution is not symmetric.

This is most relevant for count data, which are typically skewed. In extreme cases (say, Poisson distributed sales with a mean below $\log 2\approx 0.69$), your MAE will be lowest for a flat zero forecast. See here or here or here for details.

I give some more information and an illustration in What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? That thread considers the mape, but also other error measures, and it contains links to other related threads.

---

In the end, which error measure to use really depends on your Cost of Forecast Error, i.e., which kind of error is most painful. Without looking at the actual implications of forecast errors, any discussion about "better criteria" is basically meaningless.

Measures of forecast accuracy were a big topic in the forecasting community some years back, and they still pop up now and then. One very good article to look at is Hyndman & Koehler "Another look at measures of forecast accuracy" (2006).

Finally, one alternative is to calculate full predictive densities and assess these using proper scoring-rules.

Answer 2

The advantages of using MAE instead of MSE are explained in Davydenko and Fildes (2016), see Section 3.1:

...Some authors (e.g., Zellner, 1986) argue that the criterion by which we evaluate forecasts should correspond to the criterion by which we optimise forecasts. In other words, if we optimise estimates using some given loss function, we must use the same loss function for empirical evaluation in order to find out which model is better.

Fitting a statistical model usually delivers forecasts optimal under quadratic loss. This, e.g., happens when we fit a linear regression. If our density forecast from statistical modelling is symmetric, then forecasts optimal under quadratic loss are also optimal under linear loss. But, if we stabilise the variance by log-transformations and then transform back forecasts by exponentiation, we get forecasts optimal only under linear loss. If we use another loss, we must first obtain the density forecast using a statistical model, and then adjust our estimate given our specific loss function (see examples of doing this in Goodwin, 2000).

Let’s assume we want to empirically compare two methods and find out which method is better in terms of a symmetric linear loss (since this type of loss is commonly used in modelling). If we have only one time series, it seems natural to use a mean absolute error (MAE). Also, MAE is attractive as it is simple to understand and calculate (Hyndman, 2006)...

References

Davydenko, A., & Fildes, R. (2016). Forecast Error Measures: Critical Review and Practical Recommendations. In Business Forecasting: Practical Problems and Solutions. John Wiley & Sons

Answer 3

Why not compare $RMSE = \sqrt{MSE}$ and $MAE = MAD$?

Actually,

$MAE \leq RMSE \leq \sqrt{n} MAE$ for regression models:

• lower bound: each case contributes the same absolute amount of error $e$:
  $RMSE = \sqrt{\frac{1}{n} \sum e_i^2} = \sqrt{\frac{1}{n} n e^2} = e = MAE$
• upper bound: a single case having error $e$ while all other cases have 0 error:
  $MAE = \frac{e}{n}$
  $RMSE = \sqrt{\frac{1}{n} \sum e_i^2} = \sqrt{\frac{1}{n} e^2} = \sqrt{\frac{1}{n} (n MAE)^2} = \sqrt{n} MAE$

($MAE \leq RMSE \leq \sqrt{MAE}$ for classification with partial class memberships $y_i$ and/or $\hat y_i$ are $\in [0, 1]$ -- i.e. they can actually take values in between 0 and 1).

• upper bound: here, $e_i$ is $\leq 1$, so
  $MAE = \frac{n_{wrong}}{n}$
  $RMSE = \sqrt{\frac{1}{n} \sum e_i^2} = \sqrt{\frac{1}{n} n_{wrong}} = \sqrt{MAE}$
  (This upper bound occurs for integer $n_{wrong}$, if you go for partial/fractional class membership and thus also for $e_i \in [0, 1]$, things get a bit more complicated because you need to take into account that the maximum possible error can be less than 1, and you may have a "leftover" $e_i < 1$ which both lower the upper bound a bit further.)

If the RMSE is close the MAE, you have many small deviations, if it is close to its upper bound, there are few grossly wrong predictions.