Aggregating error metrics like RMSE for multiple time series

With metrics like MAE, taking a mean is the last operation in calculating error values. So given a data set of values and forecasts for multiple series for many steps into the future, taking the mean twice is equivalent to just aggregating over the whole data set. However, the last operation of RMSE is a square root, which does not distribute over the inner mean.

My question is simple: is it proper, in the case I've described, to consider the RMSE of each set of predictions as a whole, or to average the RMSEs of individual forecasts? Or are they both valid for, say, comparing two models?

For example:

Errors
0:  0,  1,  0,  0, -1
1:  1,  1, -1, -1, -1
2:  0,  0,  0,  8,  0

RMSE for whole data set ~= 2.1756

RMSE for each series:
0:  0.63
1:  1.00
2:  3.58
mean ~= 1.74


What does make sense is averaging some kind of relative RMSE. For instance, for each series, you could run a simple benchmark forecast, like the overall mean. This gives you an $$\text{RMSE}_{\text{benchmark}}$$ in the holdout sample. Now you run the forecasting method $$A$$ you are actually interested in, which gives you an $$\text{RMSE}_{\text{A}}$$ in the holdout sample. Finally you can look at the relative error in your series,
$$\frac{\text{RMSE}_{\text{A}}}{\text{RMSE}_{\text{benchmark}}}.$$
This indicates by how much $$A$$ outperforms the benchmark in terms of RMSE. And these relative measures can be compared between different series, so you can indeed take averages.