1
$\begingroup$

In the XGBoost Documentation they specify the Gain term as: \begin{equation} Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\gamma \end{equation} Loss function is:

When choosing leaf split, split that maximises Gain can not be the same as split that minimizes loss function. In some XgBoost guides people tell that best split value is chosen by Gain maximization, others say that it is chosen by loss minimization. Is there any connection between gain and loss? And how exactly is leaf split chosen if gain and loss criteria do now show the same optimal split?

$\endgroup$
1
  • $\begingroup$ 0. Welcome to the CV.SE. 1. Nice question (+1). 2. We always use Gain. Saying we use the loss does not account for our regularisation parameters. Please see my answer below where I expand on these points further. $\endgroup$
    – usεr11852
    Commented Jan 24, 2023 at 23:30

1 Answer 1

1
$\begingroup$

Yes, there is a connection between the loss function ($L$) and the Gain ($\text{Gain}$). Saying that the "best split value is chosen by Gain maximization" vs saying that the "best split values is chosen by loss minimization" is qualitatively the same thing in the context of XGBoost.

The loss function $L$ provides with the gradients $g_i$ such that $g_i = \partial_{\hat{y}_i^{(t_i-1)}}l(y_i, \hat{y}_i^{(t_i-1)})$ where $l$ is our train loss. Following that our $\text{Gain}$ uses $G_L$ and $G_R$ which correspond to $G_j =\sum_{i\in I_j} g_i$, i.e. sum of gradient values for the set of indices of the data points assigned to our left and right leafs respectively. ($\text{Gain}$ also uses the "Hessian" values $h_i$ but that is not too pertinent to the question.) As such that we see that the $\text{Gain}$ and the loss are deeply intertwine especially given we have fixed the regularisation parameters $\lambda$ and $\gamma$ that appear in the $\text{Gain}$ calculations. Informally, referring to the main $\text{Gain}$ formula at the top of OP's question, we want $G_L$ and $G_R$ to be "large" for a good split indicating that we "learn a lot" (we have a steep gradient), while we want we $H_L$ and $H_R$ to be "small" indicating that we are moving/closing to an inflection point (i.e. a minimum in the case of a convex loss function). Notice that the parameter $\gamma$ stops us from continuing our splitting operations indiscriminately, if the overall gain is not greater than $\gamma$ we stop splitting.

To conclude: saying that we pick the split that "minimises the loss" is a bit informal but not wrong. We always pick the one that maximises the $\text{Gain}$ but for a fixed set of $\lambda$ and $\gamma$ values, the two coincide so it is "qualitatively the same thing" as mentioned in the beginning. Unless one has come across it already, the XGBoost documentation offer an excellent Introduction to Boosted Trees page with a "structure score" section that elucidates this further.

$\endgroup$
5
  • $\begingroup$ Wow! Thank you very much for great explanation! $\endgroup$
    – FedorT54
    Commented Jan 25, 2023 at 9:48
  • $\begingroup$ Cool! I am glad I could help. If this answers your question you can consider accepting it as an answer. $\endgroup$
    – usεr11852
    Commented Jan 25, 2023 at 10:22
  • $\begingroup$ Hi! Actually I have one more question: In Xgboost paper formula 6 for a tree with depth 1 and number of trees=1 shows Similarity for the node which approximates minimum loss and should be equal to logloss value of the model. However when I calculate them with hands these two values are different. Why? $\endgroup$
    – FedorT54
    Commented Jan 30, 2023 at 12:19
  • $\begingroup$ Hello... It is not immediately clear to me what you compare Eq' 6 against or how its value is derived by you. I think this is a substantiative question on its own so it would be better to make a new question about it where you show your work step-by-step. It would be easier for other people to find the question and we will have more space for answers! :) $\endgroup$
    – usεr11852
    Commented Jan 30, 2023 at 12:33
  • $\begingroup$ Opened new question at stats.stackexchange.com/questions/603639/…. Best Regards $\endgroup$
    – FedorT54
    Commented Jan 30, 2023 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.