Skip to main content
added 237 characters in body
Source Link
usεr11852
  • 46k
  • 3
  • 107
  • 166

As a final comment, the best test-mae when using the standard squared error loss (objective = "reg:squarederror") is 1.623917 so we indeed do better in both runs in terms of MAE when using objective = "reg:pseudohubererror".

As a final comment, the best test-mae when using the standard squared error loss (objective = "reg:squarederror") is 1.623917 so we indeed do better in both runs in terms of MAE when using objective = "reg:pseudohubererror".

Source Link
usεr11852
  • 46k
  • 3
  • 107
  • 166

There is no definite answer at this but I would note one major and one minor point:

  1. The major point is that: A XGBoost booster starts with a base_score. That is the initial prediction score of all instances and given an adequate number of boosting iterations has been achieved, it has relatively small effect. That said, to a hard problem where the initial prediction might be way off a reasonable starting point, the whole method might get stuck. I would suggest trying different "base scores". In the example given, entering base_score=45.0 ($45$ being a round number close to the training set's median response value here) leads to the learner starting to have reasonable learning path.

It makes our learning path to look a bit like this:

[1] train-mae:8.072581  test-mae:6.321724 
Multiple eval metrics are present. Will use test_mae for early stopping.
Will train until test_mae hasn't improved in 100 rounds.

[2] train-mae:7.651685  test-mae:5.641270 
[3] train-mae:7.228202  test-mae:5.145817 
[4] train-mae:6.848772  test-mae:4.616982 
(...)
[423]   train-mae:0.571731  test-mae:1.097401 
[424]   train-mae:0.571609  test-mae:1.097115 
Stopping. Best iteration:
[324]   train-mae:0.589210  test-mae:1.096233
  1. The minor point is that: The pseudo-Huber loss function itself is parametrised by $\delta$, this what XGBoost refers as huber_slope. The derivative of our objective function approximates a straight line with slope $\delta$ for large values of our residuals but important it also approximates $\frac{a^{2}}{2}$ for small values of our residuals. So while yes, $\delta=1$ makes our function look like MAE "a lot" for large residuals values, it is the "small residuals" that actually inform our gradient step. And $\frac{1}{2}$ might be very large value leading our learner to overshoot. This parameter on it's own is not as impactful as base_score but it can help us get lower values. In the example given, after entering base_score=45.0 we can also change huber_slope=0.1 and thus get even more competitive MAE values.

And thus our learning path to look a bit like this now:

[1] train-mae:8.406398  test-mae:7.042973 
Multiple eval metrics are present. Will use test_mae for early stopping.
Will train until test_mae hasn't improved in 100 rounds.

[2] train-mae:8.313732  test-mae:6.996540 
[3] train-mae:8.238347  test-mae:6.948215 
[4] train-mae:8.171287  test-mae:6.907307 
(...) 
[263]   train-mae:1.274389  test-mae:0.244793 
[264]   train-mae:1.270874  test-mae:0.244984 
Stopping. Best iteration:
[164]   train-mae:2.070399  test-mae:0.018089

(Notice that the initial boosting rounds have higher test-mae too as our large residuals/errors are less influential than before in those early rounds.)