# Question about Xgboost paper weights and decision-rules

Can someone please explain what the weight $w$ is doing and how it works here? I also didn't understand how $q$ transforms an $m$-dimensional vector to $T$.

Edit

Answer by usεr11852 is pretty good. I also recommend anyone referring this question to refer this detailed explanation. It should put a full stop to most of the questions on boosting.

• Welcome to the CV community. Jul 8, 2018 at 20:29
• Related, in the context of binary classification with xgboost: stats.stackexchange.com/questions/350134/…
– Sycorax
Jul 8, 2018 at 20:52
• my bad! edited the link. Jul 9, 2018 at 16:02

The basic idea is behind boosting with (regression) trees is that we are learning functions $f$ (here in the form of trees $w_{q(x)}$). The weights $w$ at the $T$ leafs of the tree are representing the prediction of the $k$-th tree. $q$ is the tree structure (e.g. that of stump with only a root node, or that of an elaborate tree going 12 levels deep); $q$ is independent of the actual boosting procedure but it can affect our ensemble's performance (that's why for example we use parameters like max_depth to control it). Through the structure defined by $q$ our $m$-dimensional $x_i$ is mapped to the leaf containing the weight $w$. Within a tree $q$, $w$ itself can take $T$ different values, where $T$ is the number of leaves in the tree. Adding up all the $w_k$ from our $K$ tree learners provides us with our final estimate $\hat{y}$.
where it is clear that to get our final prediction, we are adding up weights $w_k$ as dictated by the tree structure $q_k$.
• I am glad I could help! Regarding the side-questions: 1. It is not necessarily different; it is regression tree after all. The main difference if you like is that XGBoost always does regression and even for classification tasks it does not do hard class assignment on its leaves. 2. Regularisation is done to avoid over-fitting; $\gamma$ stops trees from being overly "bushy" by penalising the number of leaves $T$ and $\lambda$ penalises very large coefficients $w$ that would potentially dominate the final summation. Jul 9, 2018 at 18:43