How to Determine Gradient and Hessian for Custom Xgboost Functions?

Question

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python implementation. Because I want to try to predict these extreme values, excluding these data points is not an option. Some of the built-in objective and evaluation metrics commonly used for regression like RMSE and even MAE can be overly sensitive to outliers, and I think it's impacting my results.

In addition to RMSLE (which I believe should be less sensitive to these sorts of extreme values), I would like to experiment with custom objective and evaluation functions. I saw this article that I think can help me get started.

However, I don't really understand how to calculate the gradient or the hessian given an objective function. Here is my current understanding of why these functions are necessary (and please someone correct me if I'm misunderstanding).

(1) The gradient is roughly analogous (or maybe it is) to the first derivative. My understanding is that this function helps the algorithm determine if a local minimum or maximum is being approached (i.e., if the first derivative is 0, then a local min/max has been achieved).

(2) The hessian is roughly analogous to the second derivative. My understanding is that this function helps the algorithm determine if the local min/max is a minimum or a maximum (i.e., if the second derivative is positive, then a local min has been achieved).

(3) However, based on what I'm reading online, the gradient and hessian seem to be based on partial derivatives.

What I don't understand is how to go from this:

$$\frac12[\log(\text{pred}+1)-\log(\text{label}+1)]^2$$

To this:

I feel like if I have a better understanding of how the gradient and hessian were calculated for RMSLE, I'll be able to do this for other objective functions too. I've tried looking up articles about the gradient and hessian online, but I'm having a very hard time figuring out how to calculate them for RMSLE.

Can anyone help walk me through how to calculate these, or point me to some online resources that can help me learn how to calculate the gradient and hessian given a particular objective function?

Answer 1

For your statements 1), 2) and 3), yes! Although, as I think you have recognised, these are very simplistic explanations. I would advise you to look at the corresponding Wikipedia pages for the gradient and the Hessian matrix.

In short, from Wikipedia, for some $f: \mathbb{R}^{n} \rightarrow \mathbb{R}$, the gradient $\nabla f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{n}$ is defined at the the point $p=(x_1, x_2, \dots x_n)$ as $$ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1}(p) \\ \vdots \\ \frac{\partial f}{\partial x_n}(p) \\ \end{bmatrix} \, . $$

For the Hessian matrix see the following screenshot from Wikipedia:

Assuming that the functions you provide are correct, they should just be implemented functions of what is provided above. As XGBoost evaluate one row at a time think of the case where $n = 1$, i.e. $f: \mathbb{R} \rightarrow \mathbb{R}$, then we have by the chain rule that $$ \frac{d}{dx}\frac{1}{2}(\log(x + 1) - \log(y + 1))^2 = \frac{\log(x + 1) - \log(y + 1)}{x+1} \, . $$ Just take the derivative of this again with respect to $x$ to get the Hessian, in the $n=1$ case it will just be a number. The implementations you provide above do elementwise operations so there are no vector operations there in the mathematical sense if that was unclear. Also, np.log1p(x) calculates $\log(1+x)$, see here.

Hope this helps :)