5
$\begingroup$

With metrics like MAE, taking a mean is the last operation in calculating error values. So given a data set of values and forecasts for multiple series for many steps into the future, taking the mean twice is equivalent to just aggregating over the whole data set. However, the last operation of RMSE is a square root, which does not distribute over the inner mean.

My question is simple: is it proper, in the case I've described, to consider the RMSE of each set of predictions as a whole, or to average the RMSEs of individual forecasts? Or are they both valid for, say, comparing two models?

For example:

Errors
0:  0,  1,  0,  0, -1
1:  1,  1, -1, -1, -1
2:  0,  0,  0,  8,  0

RMSE for whole data set ~= 2.1756

RMSE for each series:
0:  0.63
1:  1.00
2:  3.58
mean ~= 1.74
$\endgroup$

1 Answer 1

4
$\begingroup$

It usually makes little sense to average over different series' RMSEs, since these may be on very different scales.

What does make sense is averaging some kind of relative RMSE. For instance, for each series, you could run a simple benchmark forecast, like the overall mean. This gives you an $\text{RMSE}_{\text{benchmark}}$ in the holdout sample. Now you run the forecasting method $A$ you are actually interested in, which gives you an $\text{RMSE}_{\text{A}}$ in the holdout sample. Finally you can look at the relative error in your series,

$$ \frac{\text{RMSE}_{\text{A}}}{\text{RMSE}_{\text{benchmark}}}. $$

This indicates by how much $A$ outperforms the benchmark in terms of RMSE. And these relative measures can be compared between different series, so you can indeed take averages.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. I understand the metrics would be on different scales, but I'm looking for an absolute metric regardless, not a relative one (truth be told I don't know if "absolute" is the right term). I simply want to compare the results of a dataset with one model to the results given by another model with RMSE. $\endgroup$
    – Felix
    Commented Oct 7, 2020 at 7:13
  • 1
    $\begingroup$ I do not think an absolute error measure makes sense across different series. You could calculate your models' RMSE for each series, then compare the models separately for each model and see which one's RMSE is lower more often. $\endgroup$ Commented Oct 7, 2020 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.