Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Wikipedia's geral Gradient Boosting is:
Friedman's Gradient Boosting is:
Why wikipedia's gradient boosting fit h_m through pseudo-residuals while friedman uses line 4 to fit h? My question is not about he uses least squares to fit h_m but he does that fitting B.h(x,a) to pseudo-residuals instead h(x,a) to pseudo-residuals.
Are wikipedia's gradient boosting and Friedman's Gradient Boosting the same?
Responding to your comments, and bringing it back to my original comment:
Line 4 of Friedman's algorithm can be viewed as "finding the best function $g \in \mathcal{G}$ that fits the pseudo-residuals via minimizing $\sum_i [\tilde{y}_i - g(x_i)]^2$," with $\mathcal{G}$ containing all functions of the form $g(\cdot) := \beta h(\cdot; a)$.
Once you find the optimal $g(\cdot) = \beta_m h(\cdot; a_m)$ (via the optimal $a_m$ and $\beta_m$ from Line 4), you can do line 5-6 either by throwing away $\beta_m$:
(5) $\rho_m = \arg \min_\rho \sum_{i=1}^N L(y_i, F_{m-1}(x_i) + \rho h(x_i,; a_m))$
(6)$F_m(x) = F_{m-1}(x) + \rho_m h(x; a_m)$
or by keeping $\beta_m$:
(5') $\rho'_m = \arg \min_{\rho'} \sum_{i=1}^N L(y_i, F_{m-1}(x_i) + \rho' \underbrace{\beta_m h(x_i; a_m)}_{g(x_i)})$
(6')$F_m(x) = F_{m-1}(x) + \rho'_m \underbrace{\beta_m h(x; a_m)}_{g(x)}$
These two approaches are actually equivalent since the optimizers of (5) and (5') are related by $\rho_m = \rho'_m \beta_m$. The second approach is written in a form more similar to what Wikipedia is doing, since you can write steps 4-6 using only $g$ and $\mathcal{G}$.
So basically, the formulation in Friedman's "Algorithm 1" is the same as Wikipedia's, except the class of weak learners is not $\mathcal{H} = \{h(\cdot; a) \mid a \in \mathcal{A}\}$, but rather $\mathcal{G} = \{\beta h(\cdot; a) \mid \beta \in \mathbb{R}, a \in \mathcal{A}\}$.
In the special case where $\mathcal{H}$ is closed under scaling (i.e. $h \in \mathcal{H}$ implies $\beta h \in \mathcal{H}$ for any scalar $\beta$; this is true for classes like trees), then $\mathcal{H} = \mathcal{G}$ and there is no discrepancy between Wikipedia and "Algorithm 1."
