2
$\begingroup$

I am basing my understanding of the Root Mean Squared Error on this answer.

From what I understand it averages the error between the target and the prediction. The root and square parts are for keeping the result positive.

The overall goal is to minimize the error by making it as closest to $0$ as possible.

But with a few experiments in python it seems that it is possible to skew the results by 'scaling down' the data.

Here is my experiment:

import numpy as np
target_1 = np.array([0.1, 0.2, 0.3, 0.4])
target_2 = np.array([1, 2, 3, 4])
target_3 = np.array([10, 20, 30, 40])

prediction_1 = np.array([0.15, 0.19, 0.32, 0.4])
prediction_2 = np.array([1.5, 1.9, 3.2, 4])
prediction_3 = np.array([15, 19, 32, 40])

def rmse(target, prediction):
    return np.sqrt(((target - prediction) ** 2).mean())

print(rmse(target_1, prediction_1))
print(rmse(target_2, prediction_2))
print(rmse(target_3, prediction_3))

This outputs:

0.027386127875258303
0.2738612787525831
2.7386127875258306

The only real difference is their order of magnitude. How can this be a valid performance measure when the same data can achieve different results just by being in a different order of magnitude?

$\endgroup$
7
  • 2
    $\begingroup$ This is like asking whether mean length is valid because you get different numbers if the input is in mm, cm, dm or m. Multiplying data by any constant scales the RMSE proportionately. It's a consequence of the definition of RMSE that the units of measurement are precisely the same as the original: in that respect squaring and rooting cancel. Otherwise put, it's an absolute measure. If that's not what you want, you can choose something else. By the way. there is no sense in which RMSE itself has a goal of being minimal; that is perhaps a confusion with least squares as an estimation method. $\endgroup$
    – Nick Cox
    Apr 4, 2020 at 11:51
  • 2
    $\begingroup$ Sure, other things aside low RMSE beats high RMSE: the objection is to what you wrote, which confuses scale and intent. If low RMSE were the only goal, using the outcome to predict itself is the best thing to do. More seriously, a complicated model typically has lower RMSE, but that doesn't make it better. $\endgroup$
    – Nick Cox
    Apr 4, 2020 at 12:03
  • 1
    $\begingroup$ Further, RMSE may not have much or any bearing on other estimation methods. If we were using say L1-norm, using RMSE to evaluate the fit has no clear rationale. $\endgroup$
    – Nick Cox
    Apr 4, 2020 at 12:46
  • 1
    $\begingroup$ Sorry. but I don't understand what you want here. I am just adding an extra comment; L1-norm is something you can look up. It's not central to your question. $\endgroup$
    – Nick Cox
    Apr 4, 2020 at 13:04
  • 1
    $\begingroup$ @Marcus: the response is fixed - you don't change it during model selection. You only change the model, I.e. the predictions. $\endgroup$
    – Michael M
    Apr 4, 2020 at 14:32

1 Answer 1

2
$\begingroup$

You’ve left off the units!

The first one is in, say, meters, so 0.027 m.

The second one is then in decimeters, so 0.27 dm.

The third one is then in centimeters, so 2.7 cm.

Those quantities are equal!

(Ditto for liters or volts or watts or joules or Daves.)

We typically leave off units because they are implied by the data: if we measure our response variable in meters, the RMSE is in meters. However, if you review a paper claiming improvement over Dave’s algorithm because they got an RMSE of 0.27 while Dave has an RMSE of 2.7, but Dave had his measurements in cm while they did dm, both RMSEs are the same.

You don’t compare car speeds by comparing the American Corvette in mph and the Italian Ferrari in kph, as both could travel the same speed, yet the kph number would be greater.

Moral of the story: think about units.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.