How to evaluate Smooth L1 Loss, Huber Loss, etc.?

Question

I understand popular loss functions for regression tasks are Huber Loss, MAE, MSE, etc. For purposes of evaluation, popular evaluation metrics are only MAE and MSE (can you evaluate on huber loss, and does this make sense?). So for evaluating a model trained on huber loss, I am reporting both MAE and MSE as evaluation metrics. On the other hand, I could achieve better results if I just trained on MAE and MSE respectively since that is what I am evaluating the model on.

So that brings me to my main question - if I use huber loss during training, does it make sense to report MAE and MSE?

Answer 1

Short answer: Yes, you can and should always report (test) MAE and (test) MSE (or better: RMSE for easier interpretation of the units) regardless of the loss function you used for training (fitting) the model.

Long answer: The MAE and MSE/RMSE are measured (on test data) after the model was fitted and they simply tell how far on average the predictions were from the true values, i.e. they tell you how well the model performs and have nothing to do with the training itself. These two measures will almost always coincide (i.e. if a model returns lower MAE than another, it will mostly also return lower MSE). For classification you would use something else, like accuracy or AUC. Since in-sample measurement is not useful for comparing the goodness of 2 models these measures should always be measured out-of-sample, hence the well-identifyable names: test-MAE and test-RMSE.

The loss function on the other hand, is used for actually fitting a model and it can make a big difference which one to use. It has nothing to do with the test measures above, even if it can be the same one. For example, it is well-known that minimizing squared loss or equivalently MSE yields an estimate for the mean. Minimizing absolute loss or equivalently MAE yields an estimate for the median. If the goal is to estimate, for example, the mean, then using the MSE as a loss function should yield both lower test-MAE and test-RMSE than using the MAE as a loss function.

Another (extreme) example for clarity: say we want to do quantile regression, i.e. estimate an extreme quantile of the data like the 0.99 or 0.01 quantile and not the mean nor the median. Then, if we use the MAE or MSE as a loss function then they will give poorer test-MSE and test-MAE than if we used a quantile loss function (see last above). As you can see, either test evaluation metrics can be used.

In your case, Huber loss is a differentiable approximation for MAE, so I believe (without having further looked into it) that you should be getting a point-estimation for the median. So getting better test-MAE or test-MSE if you did train on MAE or MSE does not hold. It all depends on the data that you are trying to estimate. (In case you are using xgboost, note that there are some weird behaviors when using huber loss for xgboost)