I have a regression problem where I want to minimize MAE as a business metric.
I'm using LightGBM. I initially used the default
objective function for regression problems (MSE) and used MAE as my evaluation function (
eval_func) along with early stopping.
When I switched my objective function to MAE I expected to get better results because now
eval_func would be the same.
Instead my model trained on MSE results in a validation MAE that is better than the validation MAE resulting from my model trained on MAE.
My intuition is that this must be driven by the choice of hyperparameters which are the same for both models and must be working better with MSE than MAE.
In essence I'm seeking confirmation for the intuition that if I optimize my objective with MAE, given MAE optimal hyperparameters, I should be getting better MAE validation results than optimizing MSE (with optimal MSE hyperparameters). Is that mathematically true?
And what hyperparameters do usually have to be adjusted when switching from MSE to MAE as objective function?