# Gradient boosting on a loss/objective function without second derivatives

In principle, it should be possible to build a gradient boosted tree model on a loss function that only has (nonzero) first derivatives. I've found in practice xgboost and lightgbm make heavy use of second-order information in their custom objective function API. However, lightgbm (for that matter sklearn's random forest too) apparently can train on mean absolute error directly... So if I have structured data that I'd like to build a gradient boosted tree model on, what's the best way to train such a model on a loss function that has no second derivatives? Are there any (mainstream) open source tools that would work with this?

If the answer is "no", I'd love any intuition anyone can provide on why the existing packages use Newton-Raphson iterations for gradient boosting trees as opposed to some variant of raw gradient descent (as is so common in deep learning). Maybe that can give me some intuition as to how to appropriately smooth utility functions that have no second derivatives for the purposes of training gradient boosted models.

• Lots of fun little questions there (+1). Please see my answers below, I tried to keep the post short! Aug 16, 2022 at 23:31

"what's the best way to train such a model on a loss function that has no second derivatives?" The cleanest way will be to approximate/replace that discontinuous loss function with a loss that has second derivatives. For example, for MAE we can use the Pseudo Huber loss with a small $$\alpha$$. Such a replacement would take care of any inconsistencies we might expect due to discontinuities and guarantees that our implementation with theoretical derivations. That said, if the Hessian is unavailable setting it to an identity matrix makes our Newton step equal to GD step; for that matter LightGBM does exactly that i.e. hessians[i] = 1.0f; in the implementation of RegressionL1loss. It is important thought to understand why a loss does not have second derivatives and how this affects our model's behaviour. It might have minimal impact or it might signify an important transition. LightGBM developers obviously investigated this before doing such a change (link to relevant issue: here).