1
$\begingroup$

RMSE is an error metric in which the mean of the data minimizes its loss function:

$\text{RMSE} = \sqrt{\frac{\sum_{t=1}^{n}(y_t - \hat{y_t})^2}{n}}$

But it gives the error in the unit of the data.

In order to have a percentage RMSE different alternatives exist:

  1. RMSE / $\text{max}(y_t)$
  2. RMSE / $\text{mean}(y_t)$
  3. RMSE / ($\text{max}(y_t) - \text{min}(y_t)$) #if mean is zero
  4. RMSE / $\text{sd}(y_t)$
  5. RMSE / $\text{IQR}(y_t)$

If I choose option 2, can I interpret it as simple as 'the average percentage error'? (Or should I say 'the weighted average percentage error'?) Given the fact that the interpretation of RMSE is the weighted average error.

I want to make the interpretation as simple as possible for business people.

Thanks

$\endgroup$
3
  • 2
    $\begingroup$ No you cannot, since your rmse might be arbitrary large, so you will not necessarily have a value in $[0, 1]$. Also, what if your mean of $y_t$ is zero? Maybe you can explain what exactly you are trying to achieve with such a new metric. $\endgroup$ Sep 10, 2023 at 10:01
  • $\begingroup$ Why can I not? Is it because any of the 5 vary between $(-\infty, +\infty)$? If the mean of $y_t$ is zero I could use option 3. MAPE is a very appealing error metric because it is easy to explain, and the business uses it a lot (without labeling it). However, there are many flaws. So, I want a percentage metric based on RMSE that is easy to explain to the businesses (or that their acceptance will be fast). $\endgroup$ Sep 10, 2023 at 12:38
  • 2
    $\begingroup$ (2) estimates the coefficient of variation. It's okay to divide by $n$ in the "RMSE" calculation but most often the denominator is $n-df$ where $df$ regression parameters are employed in estimating $\hat y_t.$ Calling it an "average percentage error" is a mischaracterization, because the latter is English for $(100/n)\sum |y_t-\hat y_t|/y_t.$ Either concept is problematic when any of the $y_t$ might be non-positive. $\endgroup$
    – whuber
    Sep 10, 2023 at 14:29

1 Answer 1

1
$\begingroup$

First off, I would be very careful about interpreting the RMSE as a "weighted average error". Yes, larger errors add more to the RMSE than smaller ones... But it's simply not the case that the RMSE is a weighted sum of some quantities that are errors in themselves. (The wMAPE, in contrast, can be so interpreted, Kolassa & Schütz, 2007, but it has its own issues.)

Second, you can certainly scale the RMSE by some quantity and express the result as a percentage. It would then be a percentage of or with respect to that scaling denominator. As long as your denominator is positive, this is at least well-defined, but it may not give you the kind of information you or your audience needs.

I see such "scaled" RMSEs more often referred to as, well, a "scaled RMSE".

As identified in the comments, all your proposed candidates can result in errors that exceed 1, or 100% - just like the standard Mean Absolute Percentage Error can exceed 100%. This is often disconcerting to a non-technical audience. For the standard MAPE, this can be avoided by using a flat zero forecast (which is usually no use to anybody).

Forecast accuracy measurement is unfortunately harder than it looks, especially if the "accuracy" is used as a target to be "improved", which can lead to all kinds of gaming - like setting forecasts to a hard zero per above. Forecasters have to educate their audience not to over-interpret errors, and to focus more on the implications of forecast errors (e.g., Kolassa, 2023). Yes, this is often hard, and very unintuitive.

You may already have seen this: What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

$\endgroup$
2
  • $\begingroup$ What do you mean by "like setting forecasts to a hard zero per above"? My naive interpretation is that you set y_hat = 0. But I don't think that it is it... $\endgroup$ Sep 12, 2023 at 18:22
  • $\begingroup$ Oh yes indeed, that is what I meant. Assume you want to point forecast a gamma distributed variable with shape parameter $0<k\leq 1$, in a way that you minimize the expected MAPE. (Thus your realizations will be strictly positive, so you won't be dividing by zero in calculating the MAPE.) You will find that the best point forecast is a flat zero. Don't take my word for it; simulate a bunch of iid gamma variates with a shape parameter less than one, and check the MAPE for various possible point forecasts. $\endgroup$ Sep 12, 2023 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.