I want to use the Gradient Boosting algorithm with exponential loss function and I am struggling to understand how to use the Newton-Raphson update step for predictions. In python's sklearn GradientBoostingClassifier the update step is the following:
numerator = np.sum(y_ * sample_weight * np.exp(-y_ * pred))
denominator = np.sum(sample_weight * np.exp(-y_ * pred))
# prevents overflow and division by zero
if abs(denominator) < 1e-150:
tree.value[leaf, 0, 0] = 0.0
else:
tree.value[leaf, 0, 0] = numerator / denominator
The numerator is the negative sum of the partial first derivatives of the exponential loss function $$ \ L(pred) = \sum(e^{-y \cdot pred}), \text{ and } \\ \ numerator = \sum\left(-\frac{\partial}{\partial pred} L(pred)\right) = \sum(y \cdot e^{-y \cdot pred}). $$ The denominator is the sum of the partial second derivatives of the exponential loss function \begin{align} \ denominator = \sum \left( \frac{\partial^2}{\partial pred^2} L(pred)\right) = \sum(y^2 \cdot e^{-y \cdot pred}) = \sum(e^{-y \cdot pred}), \text{ since } y^2 = 1. \end{align}
But according to Newton-Raphson algorithm the update in pred should be:
\begin{align} \ pred = pred - Hessian(L(pred))^{-1} \cdot \nabla L(pred) \\ \end{align}
where Hessian is the matrix of second-order partial derivatives.
Why does Python sums over the gradient and over the Hessian and then take the ratio of the two as the update step in predictions?
As reference for the Newton-Raphson Algorithm I've been reading the following links: https://www.stat.washington.edu/adobra/classes/536/Files/week1/newtonfull.pdf https://math.stackexchange.com/questions/1153655/newtons-method-vs-gradient-descent-with-exact-line-search