From Xgboost paper

Can someone please explain what the weight $w$ is doing and how it works here? I also didn't understand how $q$ transforms an $m$-dimensional vector to $T$.


Answer by usεr11852 is pretty good. I also recommend anyone referring this question to refer this detailed explanation. It should put a full stop to most of the questions on boosting.


1 Answer 1


The basic idea is behind boosting with (regression) trees is that we are learning functions $f$ (here in the form of trees $w_{q(x)}$). The weights $w$ at the $T$ leafs of the tree are representing the prediction of the $k$-th tree. $q$ is the tree structure (e.g. that of stump with only a root node, or that of an elaborate tree going 12 levels deep); $q$ is independent of the actual boosting procedure but it can affect our ensemble's performance (that's why for example we use parameters like max_depth to control it). Through the structure defined by $q$ our $m$-dimensional $x_i$ is mapped to the leaf containing the weight $w$. Within a tree $q$, $w$ itself can take $T$ different values, where $T$ is the number of leaves in the tree. Adding up all the $w_k$ from our $K$ tree learners provides us with our final estimate $\hat{y}$.

If it helps we can continue expanding Equation 1 as: \begin{align} \hat{y}_i = \sum_{k=1}^K w_{q_k(x_i)}, \quad w_q \in R^{T_k} \end{align}

where it is clear that to get our final prediction, we are adding up weights $w_k$ as dictated by the tree structure $q_k$.

Finally, please note that while we are examining regression trees, based on the objective function we try to minimize we are able to use regression trees for many diverse tasks. For example using logistic loss lends itself naturally to classification tasks and discounted cumulative gain leads to ranking applications.

  • $\begingroup$ Thanks for an elaborate explanation. I am still wondering how is it different from the prediction of a normal decision tree? Also, what is the intuition behind including predictions(w's) in the regularization term of the loss function? Ω(f) = γT + 1 2 λkwk2 $\endgroup$ Jul 8, 2018 at 23:21
  • 1
    $\begingroup$ I am glad I could help! Regarding the side-questions: 1. It is not necessarily different; it is regression tree after all. The main difference if you like is that XGBoost always does regression and even for classification tasks it does not do hard class assignment on its leaves. 2. Regularisation is done to avoid over-fitting; $\gamma$ stops trees from being overly "bushy" by penalising the number of leaves $T$ and $\lambda$ penalises very large coefficients $w$ that would potentially dominate the final summation. $\endgroup$
    – usεr11852
    Jul 9, 2018 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.