There is no definite answer at this but I would note one major and one minor point:
- The major point is that: A XGBoost booster starts with a
base_score
. That is the initial prediction score of all instances and given an adequate number of boosting iterations has been achieved, it has relatively small effect. That said, to a hard problem where the initial prediction might be way off a reasonable starting point, the whole method might get stuck. I would suggest trying different "base scores". In the example given, entering base_score=45.0
($45$ being a round number close to the training set's median response value here) leads to the learner starting to have reasonable learning path.
It makes our learning path to look a bit like this:
[1] train-mae:8.072581 test-mae:6.321724
Multiple eval metrics are present. Will use test_mae for early stopping.
Will train until test_mae hasn't improved in 100 rounds.
[2] train-mae:7.651685 test-mae:5.641270
[3] train-mae:7.228202 test-mae:5.145817
[4] train-mae:6.848772 test-mae:4.616982
(...)
[423] train-mae:0.571731 test-mae:1.097401
[424] train-mae:0.571609 test-mae:1.097115
Stopping. Best iteration:
[324] train-mae:0.589210 test-mae:1.096233
- The minor point is that: The pseudo-Huber loss function itself is parametrised by $\delta$, this what XGBoost refers as
huber_slope
. The derivative of our objective function approximates a straight line with slope $\delta$ for large values of our residuals but important it also approximates $\frac{a^{2}}{2}$ for small values of our residuals. So while yes, $\delta=1$ makes our function look like MAE "a lot" for large residuals values, it is the "small residuals" that actually inform our gradient step. And $\frac{1}{2}$ might be very large value leading our learner to overshoot. This parameter on it's own is not as impactful as base_score
but it can help us get lower values. In the example given, after entering base_score=45.0
we can also change huber_slope=0.1
and thus get even more competitive MAE values.
And thus our learning path to look a bit like this now:
[1] train-mae:8.406398 test-mae:7.042973
Multiple eval metrics are present. Will use test_mae for early stopping.
Will train until test_mae hasn't improved in 100 rounds.
[2] train-mae:8.313732 test-mae:6.996540
[3] train-mae:8.238347 test-mae:6.948215
[4] train-mae:8.171287 test-mae:6.907307
(...)
[263] train-mae:1.274389 test-mae:0.244793
[264] train-mae:1.270874 test-mae:0.244984
Stopping. Best iteration:
[164] train-mae:2.070399 test-mae:0.018089
(Notice that the initial boosting rounds have higher test-mae
too as our large residuals/errors are less influential than before in those early rounds.)