Min_gain_split: Random Forest vs Gradient Boosting

Question

In Chapter 8 of Intro to Statistical Learning, the authors state with regard to pruning a random forest that: "One possible alternative to the process described above is to build the tree only so long as the decrease in the RSS due to each split exceeds some (high) threshold. This strategy will result in smaller trees, but is too short-sighted since a seemingly worthless split early on in the tree might be followed by a very good split—that is, a split that leads to a large reduction in RSS later on." This definitely makes sense in the context of a random forest where we are unconcerned with the variance of any given tree, but I am wondering if the same logic should be applied to gradient boosted trees.

In my case, I'm using LightGBM for a learning task and I am deciding whether to tune num_leaves or min_gain_split. I don't want to tune both, as this will increase the size of the hyperparameter space to search over (with limited benefit, I presume).

Answer 1

This same logic is partially applicable to gradient boosting regression trees (GBRTs).

While the basic notions presented are indeed pertinent for GBRTs as well, the basic reason why it isn't such a big problem is that if indeed a "very good split" is to be followed, we will find it with one of the subsequent learners. In contrast with random forests (RFs), where each base learner (tree) is oblivious to what other learners in the ensemble are doing, GBRT's base learners are aware of the residuals/gradients from the previous iterations/boosting rounds.

That said, tuning num_leaves vs min_gain_split touches upon a slightly different point: computation speed and tree-depth. On the one side, num_leaves is a very "cheap" to compute and enforce metric; it will also indirectly and lightly constrain our overall tree depth (we can still have some loop-sided trees that get quite deep asymmetrically- I assume we are growing our trees leaf-wise instead of depth-wise) and it is very objective in the sense that it does not depend on the actual task, scale of data, etc. On the other side, min_gain_split requires quite a bit of work to be computed (see the recent CV.SE thread: Gain vs Loss in terms of selecting best leaf split value for more details), it theoretically allows for "huge" base-learners (but that almost never happens in reality) and the actual value of it ($\gamma$ in XGBoost-speak) is task- & dataset- dependent so some subjectivity comes into play there.

I personally set num_leaves at a high value (e.g. 255), max_depth to be proximal value to it (e.g. 10), and optimise against min_gain_split (and min_data_in_leaf) as it guarantees shallower trees and combats over-fitting quite forcefully. It also gives me an idea of how much each learner can actually contribute. I have found though that if computational time is an issue num_leaves (with min_data_in_leaf again) allows for faster iterations and constraints the training time more aggressively. I don't think there is "one-size-fits-all" answer to this; as you correctly recognise both parameters regularise the individual base learner in a pruning-like manner. Maybe start with num_leaves as it is harder to mess up a lot and move forward with min_gain_split in a second round of modelling when some baselines are more clear.