# Gain vs Loss in terms of selecting best leaf split value

In the XGBoost Documentation they specify the Gain term as: $$$$Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\gamma$$$$ Loss function is:

When choosing leaf split, split that maximises Gain can not be the same as split that minimizes loss function. In some XgBoost guides people tell that best split value is chosen by Gain maximization, others say that it is chosen by loss minimization. Is there any connection between gain and loss? And how exactly is leaf split chosen if gain and loss criteria do now show the same optimal split?

• 0. Welcome to the CV.SE. 1. Nice question (+1). 2. We always use Gain. Saying we use the loss does not account for our regularisation parameters. Please see my answer below where I expand on these points further. Commented Jan 24, 2023 at 23:30

Yes, there is a connection between the loss function ($$L$$) and the Gain ($$\text{Gain}$$). Saying that the "best split value is chosen by Gain maximization" vs saying that the "best split values is chosen by loss minimization" is qualitatively the same thing in the context of XGBoost.
The loss function $$L$$ provides with the gradients $$g_i$$ such that $$g_i = \partial_{\hat{y}_i^{(t_i-1)}}l(y_i, \hat{y}_i^{(t_i-1)})$$ where $$l$$ is our train loss. Following that our $$\text{Gain}$$ uses $$G_L$$ and $$G_R$$ which correspond to $$G_j =\sum_{i\in I_j} g_i$$, i.e. sum of gradient values for the set of indices of the data points assigned to our left and right leafs respectively. ($$\text{Gain}$$ also uses the "Hessian" values $$h_i$$ but that is not too pertinent to the question.) As such that we see that the $$\text{Gain}$$ and the loss are deeply intertwine especially given we have fixed the regularisation parameters $$\lambda$$ and $$\gamma$$ that appear in the $$\text{Gain}$$ calculations. Informally, referring to the main $$\text{Gain}$$ formula at the top of OP's question, we want $$G_L$$ and $$G_R$$ to be "large" for a good split indicating that we "learn a lot" (we have a steep gradient), while we want we $$H_L$$ and $$H_R$$ to be "small" indicating that we are moving/closing to an inflection point (i.e. a minimum in the case of a convex loss function). Notice that the parameter $$\gamma$$ stops us from continuing our splitting operations indiscriminately, if the overall gain is not greater than $$\gamma$$ we stop splitting.
To conclude: saying that we pick the split that "minimises the loss" is a bit informal but not wrong. We always pick the one that maximises the $$\text{Gain}$$ but for a fixed set of $$\lambda$$ and $$\gamma$$ values, the two coincide so it is "qualitatively the same thing" as mentioned in the beginning. Unless one has come across it already, the XGBoost documentation offer an excellent Introduction to Boosted Trees page with a "structure score" section that elucidates this further.