Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Can someone please explain what the weight $w$ is doing and how it works here? I also didn't understand how $q$ transforms an $m$-dimensional vector to $T$.
Edit
Answer by usεr11852 is pretty good. I also recommend anyone referring this question to refer this detailed explanation. It should put a full stop to most of the questions on boosting.
The basic idea is behind boosting with (regression) trees is that we are learning functions $f$ (here in the form of trees $w_{q(x)}$). The weights $w$ at the $T$ leafs of the tree are representing the prediction of the $k$-th tree.
$q$ is the tree structure (e.g. that of stump with only a root node, or that of an elaborate tree going 12 levels deep); $q$ is independent of the actual boosting procedure but it can affect our ensemble's performance (that's why for example we use parameters like max_depth to control it).
Through the structure defined by $q$ our $m$-dimensional $x_i$ is mapped to the leaf containing the weight $w$. Within a tree $q$, $w$ itself can take $T$ different values, where $T$ is the number of leaves in the tree. Adding up all the $w_k$ from our $K$ tree learners provides us with our final estimate $\hat{y}$.
If it helps we can continue expanding Equation 1 as: \begin{align} \hat{y}_i = \sum_{k=1}^K w_{q_k(x_i)}, \quad w_q \in R^{T_k} \end{align}
where it is clear that to get our final prediction, we are adding up weights $w_k$ as dictated by the tree structure $q_k$.
Finally, please note that while we are examining regression trees, based on the objective function we try to minimize we are able to use regression trees for many diverse tasks. For example using logistic loss lends itself naturally to classification tasks and discounted cumulative gain leads to ranking applications.
