# BerHu custom loss function for XGBoost

I would like a loss function that penalizes outliers like the squared loss, while treating small errors less sharply, like the absolute loss. It seems that I am looking for a Huber loss function, but in reverse, the so-called BerHu (providing the paper in which I first saw it used, you can find it in the section 3.2. Loss Function): $$\mathrm{BerHu}_{\delta}(x)=\begin{cases} |x|, \text{ if } x\le \delta \\ \frac{x^2 + \delta^2}{2\delta}, \text{ if } x> \delta \end{cases}$$ I would like to implement this as a custom loss for an XGBoost . For this implementation, if I am not mistaken, you need to provide the gradient and hessian for the loss. However, BerHu is not twice-differentiable, since it contains the absolute value term.
I assume that it is for this reason that XGBoost does not even have an implementation for the Huber loss, instead using the Pseudo-Huber loss.
Essentially I have 2 questions:

1. What is the best way to try and implement the BerHu loss function for XGBoost? Perhaps I am wrong and there is a way to implement it exactly as is.
2. In case it is not possible, how would the analogue for a "Pseudo-BerHu" look like? I do not really understand how the Pseudo-Huber loss approximates the Huber loss, so I am struggling to come up with the analogue.

Would appreciate any suggestions on this matter. If there already exists another function that serves the same purpose, which is twice-differentiable, then I would also love to know about it

Looking at the pseudo-Huber loss, given by (to use R code):

pseudo_huber <- function(x, delta) delta^2 * (sqrt(1+(x/delta)^2) -1)


you can see how that gets the quadratic behavior near 0 and linear away from zero:

curve(pseudo_huber(x, 1), -5, 5, lwd=3)
curve(abs(x), -5, 5, add=TRUE, col="blue", lty=2, lwd=2)
curve(x^2, -5, 5, add=TRUE, col="red", lty=2, lwd=2)


Now, one idea is to just use pseudo-Huber with a really small delta to have something well-behaved and more or less linear near 0 (except for really, really close to zero, where it's more quadratic), and to add a to-the-power-4 term in the square root. E.g. like this:

pseudo_berhu <- function(x, delta, eta) (sqrt(1+(x/delta)^2 + (x/eta)^4) -1)


That (in black) certainly behaves sort of linearly near 0 (linear curve shown in blue) and quadratic (shown in red) away from it:

curve(pseudo_berhu(x, 0.1, 1), lwd=4, -20, 20, ylab="f(x)")
curve(x^2, -20, 20, add=TRUE, col="red", lty=2, lwd=3)
curve(abs(x)*10, -20, 20, add=TRUE, col="blue", lty=2, lwd=3)


Now, you'd just need to a factor to make it match the BerHu function you want.

I'm sure one can approach this differently.

Especially the second derivative gets a little messy, but exists and is not constant.

• Just to be clear, are you suggesting that I can get by with something similar to the Pseudo-Huber, except modifying it to: $\sqrt{1+\left(\frac{x}{\delta}\right)^2 + \left(\frac{x}{\eta}\right)^4}-1$? And is there a way to try and make it so that I can choose with these parameters where on the x-axis the 'split' occurs? I tried playing around with this formula in Desmos and one way to align this function with $|x|$ and not $10|x|$, like you show, would be dividing by $\eta^2$ and increasing $\eta$, but then the growth is no longer quadratic Commented May 17 at 19:24
• Yes, that's what I'm suggesting. You will not necessarily align it perfectly with something, but more important is what happens on the log-scale (ignoring what becomes additive constant there). Commented May 20 at 20:39