The Hessian in XGBoost loss function doesn't look like a square matrix

Question

I am following the tutorial for a custom loss function here. I can follow along with the math for the gradient and hessian, where you just take derivatives with respect to y_pred. The gradient is supposed to be a vector, which I can see it is, since y_pred and y are vectors. However I was assuming, since the Hessian is the Jacobian of the gradient, that the Hessian should be a square matrix, however in the definition of the tutorial it looks like just another vector, again which we can derive from the gradient by taking the derivative with respect to y_pred.

I just wanted to ask this question because I am preparing to create my own custom objective function for the first time, and I need to create these gradient and Hessian calculations. From theory I was expecting the Hessian to be a square matrix (a Jacobian) but in this tutorial it doesn't seem to be. Below is the code for calculating the Hessian. Please correct me if this in fact returns a square matrix.

def hessian(predt: np.ndarray, dtrain: xgb.DMatrix) -> np.ndarray:
    '''Compute the hessian for squared log error.'''
    y = dtrain.get_label()
    return ((-np.log1p(predt) + np.log1p(y) + 1) /
            np.power(predt + 1, 2))

Answer 1

XGBoost uses a diagonal approximation to the Hessian. A diagonal $n\times n$ matrix has at most $n$ nonzero elements. The diagonal approximation scales nicely, because it only grows linearly in $n$, as opposed to the dense Hessian which grows quadratically.

The diagonal approximation is the best when the off-diagonal elements are close to zero. However, even when the off-diagonal elements are far from zero, the reasoning is that this is tolerable because the purpose of subsequent boosting rounds is to reduce model misfit. (For an alternative perspective, consider that another group of researchers proposes a modified diagonal approximation and found that this yields better results. "A Fast Sampling Gradient Tree Boosting Framework" by Daniel Chao Zhou, Zhongming Jin, Tong Zhang.)