I have a regression problem where I want to minimize MAE as a business metric. I'm using LightGBM. I initially used the default objective function for regression problems (MSE) and used MAE as my evaluation function (eval_func) along with early stopping.

When I switched my objective function to MAE I expected to get better results because now objective and eval_func would be the same.

Instead my model trained on MSE results in a validation MAE that is better than the validation MAE resulting from my model trained on MAE.

My intuition is that this must be driven by the choice of hyperparameters which are the same for both models and must be working better with MSE than MAE.

In essence I'm seeking confirmation for the intuition that if I optimize my objective with MAE, given MAE optimal hyperparameters, I should be getting better MAE validation results than optimizing MSE (with optimal MSE hyperparameters). Is that mathematically true?

And what hyperparameters do usually have to be adjusted when switching from MSE to MAE as objective function?

  • $\begingroup$ Is the difference statistically significant? This phenomenon can easily be explained by just suggesting that we over-fitted during training. $\endgroup$ – usεr11852 Jul 11 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.