There is no definite answer at this but I would note one major and one minor point: 1. The major point is that: A XGBoost booster starts with a `base_score`. That is the initial prediction score of all instances and given an adequate number of boosting iterations has been achieved, it has relatively small effect. That said, to a hard problem where the initial prediction might be way off a reasonable starting point, the whole method might get stuck. I would suggest trying different "base scores". In the example given, entering `base_score=45.0` ($45$ being a round number close to the training set's median response value here) leads to the learner starting to have reasonable learning path. It makes our learning path to look a bit like this: [1] train-mae:8.072581 test-mae:6.321724 Multiple eval metrics are present. Will use test_mae for early stopping. Will train until test_mae hasn't improved in 100 rounds. [2] train-mae:7.651685 test-mae:5.641270 [3] train-mae:7.228202 test-mae:5.145817 [4] train-mae:6.848772 test-mae:4.616982 (...) [423] train-mae:0.571731 test-mae:1.097401 [424] train-mae:0.571609 test-mae:1.097115 Stopping. Best iteration: [324] train-mae:0.589210 test-mae:1.096233 2. The minor point is that: The [pseudo-Huber loss function](https://en.wikipedia.org/wiki/Huber_loss#Pseudo-Huber_loss_function) itself is parametrised by $\delta$, this what XGBoost refers as `huber_slope`. The derivative of our objective function approximates a straight line with slope $\delta$ for large values of our residuals but important it also approximates $\frac{a^{2}}{2}$ for small values of our residuals. So while yes, $\delta=1$ makes our function look like MAE "a lot" for large residuals values, it is the "small residuals" that actually inform our gradient step. And $\frac{1}{2}$ might be very large value leading our learner to overshoot. This parameter on it's own is not as impactful as `base_score` but it can help us get lower values. In the example given, after entering `base_score=45.0` we can also change `huber_slope=0.1` and thus get even more competitive MAE values. And thus our learning path to look a bit like this now: [1] train-mae:8.406398 test-mae:7.042973 Multiple eval metrics are present. Will use test_mae for early stopping. Will train until test_mae hasn't improved in 100 rounds. [2] train-mae:8.313732 test-mae:6.996540 [3] train-mae:8.238347 test-mae:6.948215 [4] train-mae:8.171287 test-mae:6.907307 (...) [263] train-mae:1.274389 test-mae:0.244793 [264] train-mae:1.270874 test-mae:0.244984 Stopping. Best iteration: [164] train-mae:2.070399 test-mae:0.018089 (Notice that the initial boosting rounds have higher `test-mae` too as our large residuals/errors are less influential than before in those early rounds.) As a final comment, the best `test-mae` when using the standard squared error loss (`objective = "reg:squarederror"`) is `1.623917` so we indeed do better in both runs in terms of MAE when using `objective = "reg:pseudohubererror"`.