Huber loss function is widely used, because it combines the good properties of squared and absolute losses. Therefore, when I apply the penalized regressions, i.e. LASSO, Elastic net and Ridge, to make predictions, the Huber loss is used to tune the hyperparameter by cross validation method in training process, and the MAE or MSE is applied for evaluation in validation and test stages. In Kolassa(2020), the author calims that it makes no sense that a model to be fitted by minimizing the in-sample MSE, but for holdout forecasts to be evaluated using the MAPE. (see the third point in Section 4 "Takeways").

So my main question is - Does it make sense that I use Huber loss for training, but use other measures, such as MAE and MSE, to evaluate the forecasts?

  • $\begingroup$ Might this be helpful? stats.stackexchange.com/questions/470626/… $\endgroup$ Mar 22, 2022 at 7:43
  • $\begingroup$ Why would you like to do that? $\endgroup$
    – Tim
    Mar 22, 2022 at 7:59
  • $\begingroup$ @RichardHardy, thanks for your comment. This link provides explict clarification for estimation loss and prediction loss. What I concern is the mismatch of loss function in training and evaluation process. About this question, I have looked through a lot of answers in the StackExchange website, but related answers are not that consistent. $\endgroup$ Mar 22, 2022 at 8:55
  • $\begingroup$ According this link: stats.stackexchange.com/a/518526/351802. The answerer thinks the measure for for fitting has nothing to do with the test measure. $\endgroup$ Mar 22, 2022 at 9:01
  • $\begingroup$ It would be great to disambiguate as much as possible. Estimation loss and training loss are used as synonyms in my link. How does your definition of training loss differ from estimation loss? Are you perhaps referring to evolution loss in the validation phase vs. evaluation loss in the test phase? $\endgroup$ Mar 22, 2022 at 10:04


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.