1
$\begingroup$

From my understanding of GBM is that it can take not only decision trees as base learners, but different weak learners as well (e.g., linear models), since it relies on sequential gradient descent. But what about XGBoost and LightGBM? All explanations of their methodology speak about trees form what I seen. Can't they use other weak learners as well? For context information, I need to intuitively describe their methodology in my work without getting deep into formulas and maths.

$\endgroup$
3
$\begingroup$

Yes, all GBM implementations can use linear models as base learners. In the case of XGBoost we can them directly by setting the relevant booster type parameter as being as gblinear. LightGBM does not allow for this functionality (but it has an argument lineartree that is more akin to the Cubist (or M5) model where a tree is grown where the terminal leaves contain linear regression models).

Please note that a GBM using only linear learners will essentially create a single linear regularised model. I have expanded a bit more on this matter in the thread: linear weak learners for Xgboost.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.