Why is there no improvement when training Xgboost with pseudo-Huber loss?

Question

In this StackOverflow post I asked if there was something wrong with my syntax when training an XGboost model (in R) with the native pseudo-Huber loss reg:pseudohubererror, since nor training or test error improve (remain constant). There doesn't seem to be a syntax error since custom objectives such as log-cosh loss also shows the same effect.

I am interested in understanding why it doesn't work, since training with absolute loss is a fairly popular thing, due to insensitivity to outliers, and could therefore do better than squared loss - and it is a native argument, so it must be good for something, right?. Does it have something to do with the fact that xgboost requires both a gradient and a hessian? In what context (datatype), if at all, would it work?

So far I couldn't find any example where xgboost with huber-loss is used in a concrete learning problem.

---

Here's the code from the post above as reference:

Code:

    library(xgboost)
    n = 1000
    X = cbind(runif(n,10,20), runif(n,0,10))
    y = X %*% c(2,3) + rnorm(n,0,1)
    
    train = xgb.DMatrix(data  = X[-n,],
                        label = y[-n])
    
    test = xgb.DMatrix(data   = t(as.matrix(X[n,])),
                       label = y[n]) 
    
    watchlist = list(train = train, test = test)
    
    xbg_test = xgb.train(data = train, objective = "reg:pseudohubererror", eval_metric = "mae", watchlist = watchlist, gamma = 1, eta = 0.01, nrounds = 10000, early_stopping_rounds = 100)

Result:

    [1] train-mae:44.372692 test-mae:33.085709 
    Multiple eval metrics are present. Will use test_mae for early stopping.
    Will train until test_mae hasn't improved in 100 rounds.
    
    [2] train-mae:44.372692 test-mae:33.085709 
    [3] train-mae:44.372688 test-mae:33.085709 
    [4] train-mae:44.372688 test-mae:33.085709 
    [5] train-mae:44.372688 test-mae:33.085709 
    [6] train-mae:44.372688 test-mae:33.085709 
    [7] train-mae:44.372688 test-mae:33.085709 
    [8] train-mae:44.372688 test-mae:33.085709 
    [9] train-mae:44.372688 test-mae:33.085709 
    [10]    train-mae:44.372692 test-mae:33.085709 

Answer 1

There is no definite answer at this but I would note one major and one minor point:

1. The major point is that: A XGBoost booster starts with a base_score. That is the initial prediction score of all instances and given an adequate number of boosting iterations has been achieved, it has relatively small effect. That said, to a hard problem where the initial prediction might be way off a reasonable starting point, the whole method might get stuck. I would suggest trying different "base scores". In the example given, entering base_score=45.0 ($45$ being a round number close to the training set's median response value here) leads to the learner starting to have reasonable learning path.

It makes our learning path to look a bit like this:

[1] train-mae:8.072581  test-mae:6.321724 
Multiple eval metrics are present. Will use test_mae for early stopping.
Will train until test_mae hasn't improved in 100 rounds.

[2] train-mae:7.651685  test-mae:5.641270 
[3] train-mae:7.228202  test-mae:5.145817 
[4] train-mae:6.848772  test-mae:4.616982 
(...)
[423]   train-mae:0.571731  test-mae:1.097401 
[424]   train-mae:0.571609  test-mae:1.097115 
Stopping. Best iteration:
[324]   train-mae:0.589210  test-mae:1.096233

2. The minor point is that: The pseudo-Huber loss function itself is parametrised by $\delta$, this what XGBoost refers as huber_slope. The derivative of our objective function approximates a straight line with slope $\delta$ for large values of our residuals but important it also approximates $\frac{a^{2}}{2}$ for small values of our residuals. So while yes, $\delta=1$ makes our function look like MAE "a lot" for large residuals values, it is the "small residuals" that actually inform our gradient step. And $\frac{1}{2}$ might be very large value leading our learner to overshoot. This parameter on it's own is not as impactful as base_score but it can help us get lower values. In the example given, after entering base_score=45.0 we can also change huber_slope=0.1 and thus get even more competitive MAE values.

And thus our learning path to look a bit like this now:

[1] train-mae:8.406398  test-mae:7.042973 
Multiple eval metrics are present. Will use test_mae for early stopping.
Will train until test_mae hasn't improved in 100 rounds.

[2] train-mae:8.313732  test-mae:6.996540 
[3] train-mae:8.238347  test-mae:6.948215 
[4] train-mae:8.171287  test-mae:6.907307 
(...) 
[263]   train-mae:1.274389  test-mae:0.244793 
[264]   train-mae:1.270874  test-mae:0.244984 
Stopping. Best iteration:
[164]   train-mae:2.070399  test-mae:0.018089

(Notice that the initial boosting rounds have higher test-mae too as our large residuals/errors are less influential than before in those early rounds.)

As a final comment, the best test-mae when using the standard squared error loss (objective = "reg:squarederror") is 1.623917 so we indeed do better in both runs in terms of MAE when using objective = "reg:pseudohubererror".