Metrics of Forecast Accuracy that are "fair" with respect to forecasting difficulty

Question

Let's say we want to evaluate and then compare the accuracy of forecasts across different variables for the same country, or the same variable across different countries. Are there ways to account for situations where forecasting of one indicator/country is inherently more difficult than the other, so that we can compare the forecast accuracy in a meaningful way?

• Example 1: Growth of business investment is typically more volatile than GDP growth within the same country. Both indicators are flow variables and, as such, they are more "difficult" to forecast than stock variables such as the unemployment rate.

• Example 2: GDP growth in emerging economies is typically more volatile than in big mature economies. Compare for example Turkey and the US. Moreover, in most cases, there is typically more and better data available for the big, mature economy, that the forecaster can use, which makes it easier to forecast that economy.

Note that I am not interested in model based forecast evaluation. Knowledge of how the forecasts are derived should not be required.

One strategy might be to divide both forecast and realization by the standard deviation of the realizations. Is that a valid strategy? Are there others?

Answer 1

The standard way to do this is to compare your forecast performance metric to the same metric evaluated on a benchmark model (as a ratio, say). The result is an evaluation of forecasting "skill", or what you've brought to the table above and beyond what a naïve model can do with the same data (i.e. at the same level of difficulty).

A commonly used benchmark is the random walk, but depending on context it might be fair to use a different model (e.g. white noise, or include seasonality, etc). The point is to evaluate the simplest sensible forecast that could be made without any real model selection effort. You shouldn't use a benchmark that's obviously wrong to make your own forecasts appear as great improvements, obviously. If there is a standard benchmark in the field, use that.

For both the candidate forecast and the benchmark, the forecast performance metric should be computed out-of-sample so you can have a real idea of how the candidate forecast improves performance where it matters.

Considering your proposed strategy, if the standard deviation of the realizations factors out of the performance measure in question (e.g. MAE, RMSE), your idea can be seen to be essentially equivalent to:

1. Using a white noise process as the benchmark ($Y_t = \mu + \varepsilon_t$, with $\varepsilon_t$ being iid)
2. Using in-sample performance for the benchmark
3. Using RMSE for the benchmark performance

The most problematic is point 3 because it doesn't necessarily use the same metric for the candidate forecast and the benchmark forecast, so they might not be directly comparable. Point 2 is not really an issue although the interpretation is slightly different; there are existing measures that use in-sample results for the benchmark, like MASE. Point 1 is fair, especially for growth rates of economic data. If you were looking at GDP in levels, a random walk benchmark would be more appropriate.

So, your strategy may be ok in certain contexts, but the wider point of view of comparing the same out-of-sample performance measure from the candidate forecast and a benchmark forecast will work in a wider variety of contexts.