Evaluating relative accuracy/error for continuous value prediction (and assessing relative average difference/error)

Question

I was trying to understand what are good (or standard) way of evaluating relative accuracy for continuous data.

Say for example, say I have some statistical algorithm S that outputs some real number from a vector of data. Something like $S:\mathbb{R}^k \mapsto \mathbb{R}$

Intuitively what I am interested in finding is a measure of how often S gets its approximation right. Though, since the value is not discrete, its not easy to compare it to the true value (we can't just count the times its not equal to the true value because then the accuracy would always be zero with very very very high probability).

However, we could make the algorithm S output say $n$ predictions with value $p_i$ and each has its own true value $t_i$ and then, calculate the difference from $t_i$ for each true value $t_i$. The problem is that since they all not have the same true value $t_i$, that would be maybe kind of a weird way of calculating the "average relative distance form the true value". Instead maybe calculating an average relative distance made more sense to me but was not sure if such a measure was something reasonable.

Something like:

$$\delta = \frac{1}{n}\sum^{n}_{i=1}\frac{|p_i - t_i|}{t_i}$$

If you think this is not a good metric, it would be useful to point out its weaknesses and what other metrics are better and why. Also, as n approaches infinity, does that metric approaches 0?

Please address those points in your answer (in as much detail as possible).

I was interested in doing something of that sort and was wondering what the statistics community thought about it.

---

On a side note, alternatively something that also seamed reasonable could be counting the number of times S is more than 10% (some percent) away from the true value.

Answer 1

Your $\delta$ is known in the forecasting community as the MAPE, the Mean Absolute Percentage Error. It is of course only defined if all $t_i\neq 0$, and it really only makes sense if all $t_i>0$. It has the disadvantage of penalizing overestimates much more than underestimates: the error is bounded by 100% for underforecasts, but unbounded for overforecasts. Thus, a MAPE-"optimal" prediction will be biased low.

Forecast accuracy is a frequent topic in forecasting. This link gives an overview of possible error measures.

---

Unfortunately, there is no "universally accepted" relative error measure. The MAPE (your $\delta$) is frequently used among practitioners that do time series forecasting for sales, because it has a simple interpretation, but see the weaknesses above.

A variant where we don't take the simple average of absolute percentage errors is the "weighted MAPE", where we take a weighted average, weighting each absolute percentage error by the actual:

$$\text{wMAPE} := \frac{\sum_{i=1}^n|p_i-t_i|}{\sum_{i=1}^n t_i}$$

This is still a percentage error (total absolute error as a percentage of total actuals), but it can deal with some zero actuals, and it doesn't bias the prediction as much.

The Mean Absolute Scaled Error (see the link above) is gaining traction in the academic forecasting community. It again is a relative error, giving the sum of the absolute forecast error divided by the sum of the in-sample error from some suitable benchmark method. In time series forecasting, you will usually use the naive forecast (the forecast is the last observation) as the in-sample benchmark, and occasionally use the seasonal naive forecast (the forecast is the observation from one year back). Depending on your particular prediction setting, you may be able to find a similar benchmark. The MASE then basically tells you whether your measured method out-of-sample beats the benchmark method in-sample. The MASE is still relative, but really harder to interpret, and I haven't seen any practitioners adopt it.

---

The debate about error measures has been raging for decades in the forecasting community, and it hasn't really died out because we found a solution, but because there doesn't seem to be a good one. Serious forecasters will usually assess their forecasts using multiple error measures. If forecasting method A beats method B on error measure X, but vice versa on measure Y, then there is probably not all that much to choose between the two methods.