I am following the tutorial for a custom loss function here. I can follow along with the math for the gradient and hessian, where you just take derivatives with respect to y_pred. The gradient is supposed to be a vector, which I can see it is, since y_pred and y are vectors. However I was assuming, since the Hessian is the Jacobian of the gradient, that the Hessian should be a square matrix, however in the definition of the tutorial it looks like just another vector, again which we can derive from the gradient by taking the derivative with respect to y_pred.

I just wanted to ask this question because I am preparing to create my own custom objective function for the first time, and I need to create these gradient and Hessian calculations. From theory I was expecting the Hessian to be a square matrix (a Jacobian) but in this tutorial it doesn't seem to be. Below is the code for calculating the Hessian. Please correct me if this in fact returns a square matrix.

def hessian(predt: np.ndarray, dtrain: xgb.DMatrix) -> np.ndarray:
    '''Compute the hessian for squared log error.'''
    y = dtrain.get_label()
    return ((-np.log1p(predt) + np.log1p(y) + 1) /
            np.power(predt + 1, 2))
  • 2
    $\begingroup$ The xgboost Hessian is a diagonal matrix, so if $h$ is the vector on the diagonal, we can write $hI=H$ and get a square matrix of appropriate size. $\endgroup$
    – Sycorax
    Aug 23, 2020 at 20:10
  • $\begingroup$ @Sycorax why is the Hessian assumed to be diagonal? $\endgroup$ Aug 23, 2020 at 20:15
  • 4
    $\begingroup$ A diagonal Hessian is an expedient approximation that only grows linearly with number of examples instead of quadratically. This means you gain a nice efficiency, and xgboost will attempt to correct model misfit in subsequent boosting iterations (because it's a boosting method). $\endgroup$
    – Sycorax
    Aug 23, 2020 at 20:24
  • $\begingroup$ @Sycorax OK thanks this helps answer another question I had, thinking that creating this square Hessian matrix was going to be too resource intensive for large data sets. $\endgroup$ Aug 23, 2020 at 20:28

1 Answer 1


XGBoost uses a diagonal approximation to the Hessian. A diagonal $n\times n$ matrix has at most $n$ nonzero elements. The diagonal approximation scales nicely, because it only grows linearly in $n$, as opposed to the dense Hessian which grows quadratically.

The diagonal approximation is the best when the off-diagonal elements are close to zero. However, even when the off-diagonal elements are far from zero, the reasoning is that this is tolerable because the purpose of subsequent boosting rounds is to reduce model misfit. (For an alternative perspective, consider that another group of researchers proposes a modified diagonal approximation and found that this yields better results. "A Fast Sampling Gradient Tree Boosting Framework" by Daniel Chao Zhou, Zhongming Jin, Tong Zhang.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.