What does it mean to measure model accuracy improvement?

Question

Let's say I have some data, and a model X that models the data. Let's say that model X has a RMSE value of 1500 and a MAE value of 200.

I build a new model Y, which is better than model X. Model Y has better error metrics: it has a RMSE of 1000, and a MAE of 150.

It is clear to me that model Y is a better fit than model X. But the question is: how much better?

I can calculate the percentage difference in RMSE like so: $ ({RMSE}_x - {RMSE}_y) /{RMSE}_y $. Can I say that model Y is 33% better than model X? Does this mean that my predictions are now 33% more accurate? If model Y is 33% better than model X, and we currently use model X to make decisions that results in \$100 profit, is it right to say that model Y will make \$133?

I suspect the answer is no to the above, given I could do the same to the MAE and get a different result. I understand that these questions are subjective, but if I have to quantify the improvement in model Y over model X, how would I do it?

Answer 1

1. Model Y is 33% better than X on RMSE as a criteria is the only inference you can draw.
2. Attributing profit values to model accuracy is little far fetched. Model will generally form a small (but important) part of your value chain. But predicting profit impact is a separate problem compared to lowering RMSE of prediction.
3. Model comparison based on only accuracy is also not a good practice since very complex model will help you reduce inaccuracy or errors arbitrarily. The following strategies are used for more robust model comparison,
   1. Measure accuracy on a held out / test set and not on original training data.
   2. Penalize models for their complexity, preferring simpler models. This is done by using some kind of regularization while fitting the model.

In summary, design a more evolved objective criteria to compare models which will factor in complexity of the model also. Read up more about akaike information criterion, regularization etc. if you don't know about it.

Answer 2

My typical spiel about $R^2$ seems to apply here. Below, I give a standard definition of $R^2$.

$$ R^2=1-\left(\frac{\overbrace{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2}^{\text{Square loss of your model}} }{\underbrace{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2}_{\text{Square loss of the baseline model}} }\right) $$

The fraction numerator is the square loss incurred by your model. The fraction denominator is the square loss incurred by a baseline model that always predicts the overall mean $\hat y$.

However, nothing says that we have to use such a model as the baseline to which we compare our performance. Sure, that might be a reasonable “must-beat” level of performance, but if we know our competitor to be able to achieve some level of performance and need to be able to do better than that to get any business, then that sure sounds “must beat” performance.

In fact, when you run a chunk test of nested models (such as a fairly routine ANCOVA test of a categorical variable), you are comparing the performance of your full model to a model that is reduced but perhaps not reduced all the way to predicting the same value every time, so this idea even seems to exist in the canon of standard statistical analysis!

While I would have major reservations about calling such a statistic $R^2$, it seems to be quite aligned with standard statistics ideas to calculate as below:

$$ 1-\dfrac{\text{ Performance of your model }}{\text{ Benchmark performance }} $$

I would use a similar argument about reduction in error rate (reduction in MSE, reduction in MAE, etc) as I discuss here. This relates to the Gneiting/Resin (2023) $R^*$ that I've posted about lately, such as here and here.

REFERENCE

Gneiting, Tilmann, and Johannes Resin. "Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination." Electronic Journal of Statistics 17.2 (2023): 3226-3286.