4
$\begingroup$

I am aware of gradient boosted trees. The extreme-gradient boosting algrithm is widely applied these days. What excactly is the difference between the tree booster (gbtree) and the linear booster (gblinear)?

What I understand is that the booster tree grows a tree where a fit (error rate for classification, sum-of-squares for regression) is refined taking into account the complexity of the model. What is done differently in a linear booster?

$\endgroup$
4
$\begingroup$

I just recently started using gradient boosted trees, please correct me if I'm wrong. I found this wiki page https://en.wikipedia.org/wiki/Gradient_boosting informative. Check out the algorithm and gradient tree boosting section.

As far as I understand, gradient boosting will of course work with most learners. Gradient boosting will in iterations($m$) train one new learner $h_m$ on the ensemble residuals of of previous iteration.

The ensemble $F_m$ is updated with $F_m \leftarrow F_{m-1} + \gamma_m h_m$

where $F_{m-1}$ was the previous ensemble and $\gamma_m$ is a coefficient such that,

$ \gamma_m = \underset{\gamma}{\operatorname{arg\,min}} \sum_{i=1}^n L\left(y_i, F_{m-1}(x_i) + \gamma_m h_m(x_i)\right).$

Hereby is the new learner fused with old ensemble by coefficient $\gamma_m$, such that new ensemble explains the target $y$ the most accurately(defined by loss function metric $L$)

As explained on wiki page, Friedman proposed at special modification for decision trees, where each terminal node $j$*** of the new learner $h_m$, has it own separate $\gamma_{jm}$ value. This modification would not be transferrable to most other learners, such as gblinear.

*** (wiki article describes each $\gamma_{jm}$ to cover a disjoint region(R) of the feature space. I prefer to think of its as the terminal nodes, that happens to cover each a disjoint region)

Also to mention, if you pick a strictly additive linear regression as base learner, I think the model will fail fitting interactions and non-lineairties. In example below the xgboost cannot fit $y=x_1 x_2$

library(xgboost)
X = replicate(2,rnorm(5000))
y = apply(X,1,prod)
test = sample(5000,2000)
Data = cbind(X=X)
xbm = xgboost(Data[-test,],label=y[-test],params=list(booster="gblinear"),nrounds=500)
ytest = predict(xbm,Data[test,])
plot(y[test],ytest)
$\endgroup$
  • $\begingroup$ Thanks for your comments. I have also read a lot about these methods, applied xgboost and gbm. I just wonder what "gblinear" linear is. I just saw a qora question about this today. I think it is not described in the documentation of xgboost. $\endgroup$ – Ric Mar 15 '16 at 15:47
  • 2
    $\begingroup$ It is simply multiple linear regression (MLR) without interactions, xgboost.readthedocs.org/en/latest/R-package/… $\endgroup$ – Soren Havelund Welling Mar 17 '16 at 9:57
  • $\begingroup$ I was on this page too and it does not give too many details. If I think of the approaches then there is tree boosting (adding trees) thus doing splitting procedures and there is linear regression boosting (doing regressions on the residuals and iterating this always adding a bit of learning). You are right, it could be the latter (most probably is) but I find it confusing that this is not explicitely written anywhere ... $\endgroup$ – Ric Mar 17 '16 at 10:00
  • 1
    $\begingroup$ well rather elastic net..... /*! * Copyright 2014 by Contributors * \file gblinear.cc * \brief Implementation of Linear booster, with L1/L2 regularization: Elastic Net * the update rule is parallel coordinate descent (shotgun) * \author Tianqi Chen */ github.com/dmlc/xgboost/blob/… $\endgroup$ – Soren Havelund Welling Mar 17 '16 at 11:07
  • $\begingroup$ Right .great!. the description of the L1 and L2 penelty terms that I was able to find could have made me think of this. If you post this as answer then I would accept. In short. gbtree is a gradient descent of tree type with penalty on complexity and gblinear is a regression (in the sense of elastic net) boosting ... $\endgroup$ – Ric Mar 17 '16 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.