5
$\begingroup$

There are several posts on here about this question. The gist of them, as far as I understand, is that you cannot compare RMSE or MAE of two models where one is log transformed on the dependent variable since they are on different scales. So you can compare thousands of dollars to thousands of dollars but not log(thousands of dollars) to just thousands of dollars.

e.g.

Huge difference in regression standard error after log transformation of dependent variable

Back transforming regression results when modeling log(y)

Linear regression with log transformed data - large error

Compare Linear and Log standard error after transformation in R

I then found this article by Duke What's the bottom line? How to compare models.

In particular, this paragraph:

The root mean squared error and mean absolute error can only be compared between models whose errors are measured in the same units

(e.g., dollars, or constant dollars, or cases of beer sold, or whatever). If one model's errors are adjusted for inflation while those of another or not, or if one model's errors are in absolute units while another's are in logged units, their error measures cannot be directly compared. In such cases, you have to convert the errors of both models into comparable units before computing the various measures. This means converting the forecasts of one model to the same units as those of the other by unlogging or undeflating (or whatever), then subtracting those forecasts from actual values to obtain errors in comparable units, then computing statistics of those errors. You cannot get the same effect by merely unlogging or undeflating the error statistics themselves!

Does this mean that a sound approach to choosing between a log model and it's equivalent non log version would be to do e.g. cross validation and pick the model with the lowest RMSE or MAE after predicting on e.g. 5 folds?

Put another way, the questions I found while researching gave descriptive reasons why you cannot compare RMSE directly with initial model output, but if I understand if I just use the model to predict on test data, I can have a somewhat definitive answer to which model to select?

Is this a logical approach? A standard or typical approach? A good approach?

Improve this question
$\endgroup$
7
  • $\begingroup$ If down voting can you let me know how to improve the question. Thanks $\endgroup$ – Doug Fir Apr 8 '17 at 10:21
  • 3
    $\begingroup$ I think there is a random down-voter at large. Sadly I cannot answer this question though but it seems useful so I up-voted it. Hope an expert passes by. $\endgroup$ – mdewey Apr 8 '17 at 12:56
  • $\begingroup$ I noticed that too, @mdewey. I've upvoted this question out of counter-spite. $\endgroup$ – The Laconic Apr 9 '17 at 1:37
  • 1
    $\begingroup$ I don't see the relevance of cross-validation here in particular. The idea is that you have to compare apples to apples, whatever your metric. This is conceptually distinct from the process of comparing the performance of two models on test data. Edit: I take that back. I can see that performance on test data is important here, since one model could be directly optimizing the performance measure in-sample, while the other wouldn't. But it does still have to be apples-to-apples. $\endgroup$ – The Laconic Apr 9 '17 at 1:41
  • $\begingroup$ @TheLaconic thanks for contributing to the discussion. You don't see the relevance of cross-validation here. OK. But I was not sure. That's why I asked a question on this q&a site... out of lack of understanding! $\endgroup$ – Doug Fir Apr 9 '17 at 1:45
2
$\begingroup$

Yes, what you describe is a logical approach.

Aside (back-)transforming the response variable I would suggest considering a model that does not rely heavily on assumptions regarding the model's error-structure and/or the distribution of the response variable. Immediate regression-like alternatives would be robust regression and quantile regression. Similarly there is little reason not to use tree-based (like CHAID trees) or gradient-boosting approaches (like XGBoost) if you are mostly interested in prediction rather than statistical inference.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks. I'll explore the other models that you suggested too. Just to confirm, when you say back transform I (r) need to exp(fitted.values(mymodel)) then manually calculate residual standard error or evaluation metric? i.e. R caret summary(mylogmodel)) residual standard error is not the comparable metric but I can compare to non log version if I back transform the fitted values and then manually calculate e.g. Residual standard error? $\endgroup$ – Doug Fir Apr 8 '17 at 23:32
  • $\begingroup$ Cool! Yes, exponentiating the $\log(\hat{y})$ will give provide estimates on the original scale of the $y$ variable. Residuals will be derived from these values. The link (you might have already seen here) has a nice example. $\endgroup$ – usεr11852 Apr 9 '17 at 10:37
  • $\begingroup$ Help! I'm receiving conflicting information over here. Is it enough to calculate new residuals and thus new RMSE for log model based on exponentiating the log(ŷ ) alone or do I have some acrobatics to do? Please see linked to crossvalidated post :-/ $\endgroup$ – Doug Fir Apr 9 '17 at 11:48
  • $\begingroup$ He said the same things as I do: "You need to transform your prediction back to the original space before calculating residuals rather than transforming the residuals from the log space." (+1 to his post by me!) $\endgroup$ – usεr11852 Apr 9 '17 at 11:56
  • $\begingroup$ I understand the source of confusion; sorry, it is because that answer uses Duan's smearing estimator while the way I commented on it makes no explicit comment on the bias. I was sloppy in my comment, apologies. See Bill's excellent answer here. $\endgroup$ – usεr11852 Apr 9 '17 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.