4
$\begingroup$

I'm working on a machine learning problem, and I'm having trouble interpreting different measures of model performance. I have a single dependent variable (proportion change between two treatments, ranges from -1 to 1 but mostly close to 0) from a dataset that I'm modeling using three approaches: 1) XGboost, 2) a simple neural network, and 3) a null model (fit to mean, "featureless" in mlr parlance). To compare the performance of the models, I've compared out-of-sample model performance across repeated random folds in the dataset, and have mean absolute error (MAE), root mean squared error (RMSE), and R-squared shown below, all for the independent testing folds of the dataset:

MLbenchmarks

As you can see, the MAE scores show that two models are no better than the null model (featureless), but the RMSE and R-squared show the opposite result (XGboost and NNet outperform the null). I've read other articles on the difference between RMSE and MAE (here, here, and here), and I'm pretty sure that this has to do with something related to the difference between the mean and median error where error is skewed or non-symmetrical (i.e. Jensen's inequality).

How should I properly interpret the results of these benchmarks? Are the XGboost and NNet models "truly" better than the null model, or am I missing something?

$\endgroup$

2 Answers 2

5
$\begingroup$

Different error metrics elicit different point predictions. That is, if you want to minimize the RMSE, you should a priori output a different prediction than if you wanted to minimize the MAE. Therefore it also makes no sense to evaluate the same point prediction using different error metrics.

There is no "truly" better. One method of calculating a point prediction may be closer to the functional elicited by one error metric (e.g., the conditional mean for the RMSE) than another method - but the second method may yield predictions closer to the conditional median and therefore yield lower MAEs.

I go into more detail in Kolassa (2020, IJF). Ping me on ResearchGate or LinkedIn if you are interested and don't have access.

Note that all these are in expectation only. Predictions are random variables, and we are always arguing from finite samples.

$\endgroup$
2
  • $\begingroup$ "Therefore it also makes no sense to evaluate the same point prediction using different error metrics." I'd disagree with that. I'd say that unless there are clear reasons why one metric is inappropriate in a given application, it is informative if and to what extent different metrics disagree. In particular, if you try to optimise RMSE, say, it is a good thing if you find out that your prediction method also does well regarding MAE. $\endgroup$ Commented May 9 at 9:55
  • $\begingroup$ @ChristianHennig: of course you are free to disagree. I personally would say that no, it is not informative, per my answer. We may have different notions of what "informative" means. I may be predisposed to this PoV because of my background in count data forecasting, where there are indeed clear reasons why MAE and MSE answer quite different questions. $\endgroup$ Commented May 9 at 11:54
2
$\begingroup$

Adding to Stephan Kolassa's answer, the most reasonable interpretation is this: Your xgboost an NN model were probably trained to give good mean predictions (e.g. if you used squared loss). The MAE is designed to measure the quality of median predictions, so these models do not fare better than a simple benchmark in this task. When you compare via RMSE and R2 though, you use metrics which are designed to compare mean predictions, thus your models can now show their potential for the task they were build to do.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.