1
$\begingroup$

When I built the model, I applied the log transformation to all variables including the dependent variables. Now, I'm calculating the RMSE for the evaluation, and the result is in the log format, which is pretty small, around 0.48.

However, I don't think the number is correct to evaluate the model under this situation. For example, if one actual value is 10.1 after log transformation, and its predicted value is 10; another actual value is 1.1, and its predicted value is 1, the residuals under the log transformation of these two numbers are both 0.1. But if I convert them back to the original format and calculate the residuals: the residuals of exp(10.1)-exp(10) is way too different from the exp(1.1)-exp(1).

My question is: How to I fix this problem? (ex.manually calculate the RMSE after converting the predicted value in log format to the original format?)

$\endgroup$
  • $\begingroup$ What was the loss function you used in your model on a log scale? $\endgroup$ – Stephan Kolassa Jun 24 '19 at 17:47
  • $\begingroup$ I'm not sure about the loss function. I used h2o.gbm() and randomForest() pacakge to run the model with the default setting.@StephanKolassa $\endgroup$ – Fangyuan Jun 24 '19 at 18:08
  • $\begingroup$ Do you have idea/reason why you applied the log transformation on dependent variable? $\endgroup$ – user158565 Jun 25 '19 at 2:41
3
$\begingroup$

Whether it makes more sense to evaluate your predictions on the original or on the log scale depends on your application, or on your loss function. Essentially: use a loss function that correctly measures how painful each loss is. (Which is two ways of saying the same thing.) We statisticians here can't tell you this. It depends on your domain.


Note that you have at least one error in your analysis.

  • If you want an unbiased prediction, then the RMSE is the correct evaluation tool, since it will be minimized (in expectation) by an unbiased prediction. However, an unbiased prediction $\hat{y}$ on the log scale turns into a biased prediction on the original scale if you simply exponentiate, $e^{\hat{y}}$. You need a bias correction term, $e^{\hat{y}+\frac{\hat{\sigma}^2}{2}}$. (Yes, this also holds for non-time series predictions.)
  • If you want a point prediction of the conditional median, then you can simply exponentiate, $e^{\hat{y}}$, to turn a median prediction on the log scale into a median prediction on the original scale (because exponentiation is monotone). However, then the RMSE is not an appropriate measure any more, at least on the original scale. You should use the MAE, which is minimized in expectation by a median prediction.

More information here and here and here.

$\endgroup$
  • $\begingroup$ Thank you so much for this explanation. I think I want an unbiased prediction because my model is to predict the cost using different categorical and continuous variables, like length and materials. Should I calculate the RMSE after I finish the back-transformation with bias correction term by myself instead of the RMSE given by the model in log scale? $\endgroup$ – Fangyuan Jun 24 '19 at 19:25
  • $\begingroup$ I address your question in the first paragraph of my answer - it depends on what your loss is. For costs, it sounds more reasonable to work on the original scale. $\endgroup$ – Stephan Kolassa Jun 25 '19 at 5:31
1
$\begingroup$

As @StephanKolassa says, this depends on your application.*

One important thing to keep in mind: log transforms make a lot of sense when errors are proportional to the values, as this page discusses. If you are primarily interested in relative (or percentage) errors, staying in the log scale might be best. In your example, the difference between 10.1 and 10.0 on (natural) log-transformed data is just over 10% in the original scale, the same percentage difference in the original scale as there is between 1.1 and 1.0 on the log-transformed scale.

Note that your RMSE in the log scale of 0.48 isn't actually small at all if the dependent variable was log-transformed; exp(0.48) = 1.62, or a 62% error!

If your interest is in errors on the original scale of measurement, however, then you will either (1) have to use the types of approaches that @StephanKolassa recommends or (2) redo the analysis in the original scale of measurement of the dependent variable. Given the 62% RMSE error from your model with all variables log-transformed, you should revisit your approach to the modeling.


*The need of your application should always inform your choice of a loss function. Look at the documentation for the packages you used to see what they choose as default loss functions if you don't specify one yourself. As it is increasingly easy to use powerful statistical tools, it's very important to know what hidden assumptions those tools are making about your data before you rely on them.

$\endgroup$
  • $\begingroup$ Thanks for your suggestion and I agree with it. I did try to find the loss function in documentation of the model I applied, but there was no exact "loss function" mentioned. The closest one I found was stopping_metric of h2o.gbm. The AUTO option elaborates Logloss for classification, deviance for regression. Is it loss function of this model? $\endgroup$ – Fangyuan Jun 24 '19 at 20:12
  • $\begingroup$ @FangyuanLi yes, the stopping_metric is the loss function; it's what the program tries to minimize. See this page for what deviance means, particularly in the context of regression trees. Deviance is useful for many types of regression; it is related to mean-square error in standard linear regression. $\endgroup$ – EdM Jun 24 '19 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.