How do you Interpret RMSLE (Root Mean Squared Logarithmic Error)?

Question

I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem is I'm not sure how to interpret the success of my final result.

For example if I achieved a RMSLE of $1.052$ could I raise it the the exponential power $e$ and interpret it like rmse? (ie. $e^{1.052}=2.863=RMSE$)?

Could I then say that my predictions were $\pm \$2.863$ on average from the the actual prices? Or is there a better way to interpret the metric? Or can the metric even be interpreted at all with the exception of comparing to the other RMSLEs of other models?

Answer 1

I haven't seen RMSLE before, but I'm assuming it's $\sqrt{ \frac{1}{N} \sum_{i=1}^N (\log(x_i) - \log(y_i))^2 }$.

Thus exponentiating it won't give you RMSE, it'll give you

$e^\sqrt{ \frac{1}{N} \sum_{i=1}^N (\log(x_i) - \log(y_i))^2 } \ne \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - y_i)^2}$.

If we take the log of both sides, we get the RMSLE versus $\frac{1}{2} \log \left( \frac{1}{N} \sum_{i=1}^N (x_i - y_i)^2 \right)$, which is clearly not the same thing.

Unfortunately, there isn't a good easy relationship in general (though someone smarter than me / thinking about it harder than me could probably use Jensen's inequality to figure out some relationship between the two).

It is, of course, the RMSE of the log-transformed variable, for what that's worth. If you want a rough sense of the spread of the distribution, you can instead get a rough sense of the spread of their logarithm, so that a RMSLE of 1.052 means that the "average" is $2.86$ times as big as the true value, or 1/2.86. Of course that's not quite what RMSE means....

Answer 2

I don't know if there is a straightforward generic interpretation even analyzing a particular case.

For example, you may be interested in evaluating what would be the error if you predict all the cases with the mean value and compare it to your approach.

Anyway, I believe that RMSLE is usually used when you don't want to penalize huge differences in the predicted and true values when both predicted and true values are huge numbers. Only the percentual differences would matter since you can rewrite the natural logarithm equation:

$\log{(P_i + 1)} - \log{(A_i +1)} = \log{\frac{P_i + 1}{A_i +1}}$.

For example, for Pi = 1,000 and Ai = 500 would give you the roughly same error with when Pi = 100,000 and Ai = 50,000.

Answer 3

There is an indirect way of measuring the performance of a loss function in terms of something more easily understandable, although it will not directly convert values as you have hoped.

Once the model has been trained and tested using RMSLE, simply take a new metric on it. Just because the model was trained on RMSLE, that doesn't mean you can't then take other more understandable loss functions as metrics.

In Keras, for example, you can specify extra loss functions in a metrics category in the model compiler. In the beneath the MSLE is used to train the model (Equivalent to the RMSLE), but the MAE and MSE are also recorded:

model.compile(loss='mean_squared_logarithmic_error', optimizer='adam', metrics=['mean_absolute_error','mean_squared_error'])

Answer 4

My understanding is, when we do logarithm both on prediction and actual numbers, we'll get much smoother results than original one. And reduce the impact of larger x, while emphasize of smaller x for $\log{x+1}$.

Also you'll get a intuitive impression by drawing a simple graph of $y=\log{x+1}$.