# The Hessian in XGBoost loss function doesn't look like a square matrix

I am following the tutorial for a custom loss function here. I can follow along with the math for the gradient and hessian, where you just take derivatives with respect to y_pred. The gradient is supposed to be a vector, which I can see it is, since y_pred and y are vectors. However I was assuming, since the Hessian is the Jacobian of the gradient, that the Hessian should be a square matrix, however in the definition of the tutorial it looks like just another vector, again which we can derive from the gradient by taking the derivative with respect to y_pred.

I just wanted to ask this question because I am preparing to create my own custom objective function for the first time, and I need to create these gradient and Hessian calculations. From theory I was expecting the Hessian to be a square matrix (a Jacobian) but in this tutorial it doesn't seem to be. Below is the code for calculating the Hessian. Please correct me if this in fact returns a square matrix.

def hessian(predt: np.ndarray, dtrain: xgb.DMatrix) -> np.ndarray:
'''Compute the hessian for squared log error.'''
y = dtrain.get_label()
return ((-np.log1p(predt) + np.log1p(y) + 1) /
np.power(predt + 1, 2))

• The xgboost Hessian is a diagonal matrix, so if $h$ is the vector on the diagonal, we can write $hI=H$ and get a square matrix of appropriate size.
– Sycorax
Aug 23, 2020 at 20:10
• @Sycorax why is the Hessian assumed to be diagonal? Aug 23, 2020 at 20:15
• A diagonal Hessian is an expedient approximation that only grows linearly with number of examples instead of quadratically. This means you gain a nice efficiency, and xgboost will attempt to correct model misfit in subsequent boosting iterations (because it's a boosting method).
– Sycorax
Aug 23, 2020 at 20:24
• @Sycorax OK thanks this helps answer another question I had, thinking that creating this square Hessian matrix was going to be too resource intensive for large data sets. Aug 23, 2020 at 20:28

XGBoost uses a diagonal approximation to the Hessian. A diagonal $$n\times n$$ matrix has at most $$n$$ nonzero elements. The diagonal approximation scales nicely, because it only grows linearly in $$n$$, as opposed to the dense Hessian which grows quadratically.