2
$\begingroup$

Let's say I have two different models of an outcome Y, m1 and m2 and perform some kind of cross-validation.

I calculate the RMSE and the MAE on the test set (for the two models) and I want to say something substantively about the change in RMSE and MAE between model 1 and model 2.

Is it feasible calculate the ratio, such as RMSE_m1/RMSE_m2 or similarly (RMSE_m1-RMSE_m2)/RMSE_m1?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I find this reasonable.

Imagine a situation where some baseline model (your company’s standard) results in performance of $50$ that is considered pretty good but certainly not perfect. This performance could be RMSE, MAE, or something else.

After making some changes to the model (say you hire someone who makes more accurate measurements of important features), your new model results in performance of $45$.

I find it reasonable to describe the new model as having made a $10\%$ improvement on performance, yes, from $50$ down to $45$.

$$ \dfrac{50-45}{50}= 1-\dfrac{45}{50} =10\% $$

This is in the spirit of how I think about $R^2$ that I’ve written about on Cross Validated many times, such as here, here, here, and here, among others. Sure, that uses the performance of what I call a “naïve baseline” in the denominator, but an important consideration is that it is some kind of “must beat” level of performance.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I agree with that statement (+1) and add two things: 1) you find the ratio of RMSE/MAEs quite broadly in the empirical forecasting literature and 2) it is quite comparable to $R^2$. The latter describes how well you fit the data in-sample, while the latter describes how well you fit the data out-of-sample. $\endgroup$
    – Louki
    Commented Oct 28 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.