1
$\begingroup$

I am evaluating two machine learning models. The output is count data which has a range of 0 to 30, which most of the output values being small values. Large output values are rare.

enter image description here

One model has lower MAE and RMSLE and the other model has lower RMSE. I am not sure which model is performing better. It is worth noting that Model 2 is the result of model after taking log transformation of the output variable.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Which car is better when they aren't equally fast, cheap or stylish? It's not even axiomatic that minimising a measure of global lack of fit is the way to choose a model. Other criteria might include simplicity; ease of interpretation; matching patterns qualitatively; and on and on and on. $\endgroup$
    – Nick Cox
    Aug 20, 2019 at 18:17
  • $\begingroup$ Thank you for your reply. I have actually transformed the logged predictions back so that the results for these two models should be on the same scale. I think they are comparable? Just want to know if results are better by taking log transformation. $\endgroup$
    – jkjsdf fod
    Aug 20, 2019 at 18:24
  • $\begingroup$ I wouldn't want to choose a model on this information alone. $\endgroup$
    – Nick Cox
    Aug 20, 2019 at 18:27
  • 2
    $\begingroup$ Did you adjust for bias when back-transforming predictions on a log scale? $\endgroup$
    – Stephan Kolassa
    Aug 20, 2019 at 22:45
  • 1
    $\begingroup$ Yes, it does. Look at the "bias adjustments" section at the bottom of the page I linked. You need to include a correction term for the predicted variance on the log scale. It's similar to the lognormal distribution: if the mean on the log scale is $\mu$, then the mean on the original scale is not $\exp(\mu)$, but $\exp(\mu+\frac{\sigma^2}{2})$. ... $\endgroup$
    – Stephan Kolassa
    Aug 21, 2019 at 6:21

1 Answer 1

Reset to default
3
$\begingroup$

Comparatively, RMSE penalizes large gaps more harshly than MAE, and RMSLE penalizes large gaps among small output-values more harshly than large gaps among large output-values (in fact, penalizes according to the ratio rather than the difference).

So, it seems that on average, Model2 makes more/bigger large-scale errors and fewer/smaller small-scale errors, and tends to make its large-scale errors in the large-output range, all compared to Model1. (And this was probably to be expected, since you fit that model on the log-transformed outputs.)

Which one is better then is up to your use-case. Given only your description (count data, primarily small values), I would personally prefer Model2, but YMMV. Perhaps ask yourself if a prediction of 3 for a true value of 2 better or roughly the same as a prediction of 18 for a true value of 12? I'd also support additional investigations, as suggested by @NickCox's comments.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for your reply. Would you recommend log transform count data for non-parametric machine learning models if the data is extremely right skewed? $\endgroup$
    – jkjsdf fod
    Aug 20, 2019 at 18:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.