Does boosting being component-wise imply variable selection?

Question

I have come across the term "component-wise" in the literature, and I am curious if this means that a model does perform variable selection. And not having this, mean it doesn't perform variable selection.

More specifically, I am currently exploring boosted trees, specifically those introduced by Friedman in his 2001 paper, "Greedy function approximation: A gradient boosting machine." Could one say it's a variable selection model? Equivalently, component-wise boosting?

I appreciate any insights or clarifications on this topic. Thank you.

Answer 1

The fitting formed by the base learners during the training of GBM acts an implicit variable selection indeed; Additionally, the component-wise nature of fitting a GBM is doing some variable selection too but that is not the primary reason why variable selection happens.

Let's clarify things a bit further: A GBM performs "component-wise" learning in the sense that each new individual base learner tries to counter-balance the mistakes of the previous iterations. This is very obvious in the case of GAMs where via the back-fitting algorithm we have component-wise smoothing splines for one selected feature $x_j$ at the time but it extends naturally to GBMs too. That said, GAMs do not perform any variable selection; they have some regularisation properties associated with the cost of the component-wise smoother but do not actually perform variable selection explicitly. A GAM might have a very flat smooth associated with a "useless" feature but that's about it. Some extensions of GAMs do perform variable selection (e.g. see Marra & Wood (2011) Practical variable selection for generalized additive models) but that's an additional step in the fitting procedure.

Revising now the case of a GBM: the base learners are trees so variable selection is performed in the sense that features that do not contribute to loss function reduction are not selected by the base learners themselves. Each individual tree performs a weak form of variable selection. In addition, the GBM itself regularises the contribution of each individual base learner based on the shrinkage/learning rate $\alpha$ further stopping a base learner's potential overfitting to harm the overall GBM's overall performance. Now, given we usually have dozens, if not hundreds of base learners in our GBM, the tree ensemble as a whole indeed performs performance variable selection via regularisation in two complementary ways: 1. within a base learner and 2. across base learners when combining them. That said, it is not the component-wise training that primarily leads this but rather fitting itself. (For example, random forests perform a similar "variable selection" procedure, BARTs even more.)