I am following the tutorial for a custom loss function here. I can follow along with the math for the gradient and hessian, where you just take derivatives with respect to
y_pred. The gradient is supposed to be a vector, which I can see it is, since
y are vectors. However I was assuming, since the Hessian is the Jacobian of the gradient, that the Hessian should be a square matrix, however in the definition of the tutorial it looks like just another vector, again which we can derive from the gradient by taking the derivative with respect to
I just wanted to ask this question because I am preparing to create my own custom objective function for the first time, and I need to create these gradient and Hessian calculations. From theory I was expecting the Hessian to be a square matrix (a Jacobian) but in this tutorial it doesn't seem to be. Below is the code for calculating the Hessian. Please correct me if this in fact returns a square matrix.
def hessian(predt: np.ndarray, dtrain: xgb.DMatrix) -> np.ndarray: '''Compute the hessian for squared log error.''' y = dtrain.get_label() return ((-np.log1p(predt) + np.log1p(y) + 1) / np.power(predt + 1, 2))