the score to hope for when evaluating model by MAE, MSE or RMSE

Question

when doing evaluation and optimization of model by MAE, MSE or RMSE what should we look at and compare our score to, as a baseline or acceptable score for our model. should we look at the best constant for metrics, for example median of target values for MAE or the mean of target values for MSE?

Answer 1

It's better not to use a function of the targets, since you don't know them when you calculate your predictions. Plus, there may be some pattern that should be easy to include in your model but which would mean that future values are non-stationary - the simplest example would be seasonality.

Instead, fit a simple benchmark model to your training data. That could be the mean of the training realizations, or the median, or the last observation (AKA a "naive" forecast), or the last observation one seasonal cycle back (a "naive seasonal" forecast), or a regression on one or two obvious drivers.

The last two examples - a naive seasonal model and a simple regression - illustrate cases where the benchmark should yield non-stationary predictions, per above.

Then calculate the KPI you are interested in on this benchmark. Finally, evaluate a possible more complex model on whether it consistently beats the benchmark.

It is often surprisingly hard to beat a simple benchmark.