Some of our BI analysts and most of our managers are interested in making explainable predictions. One of our colleagues proposed an approach based on individual tree leaves from a tree-based algorithm (that we also use as black box predictors). Preference for a collection of explicitly defined tree leaves over an unobservable ensemble of many such leaves (i.e. a tree-based model) is motivated by explainability and traditional century-old insurance industry methods of discovering data clusterings with higher average target values (typically some risk measures).


The main difficulty is how to use data partitioning information from more than a single tree, utilizing benefits of ensembling and cross-validation. Note that in practice tree structure tends to differ between all trees in the ensemble (such as boosting rounds in a GBDT algo) and between all cross-validation folds, so a leaf data usually cannot be averaged from multiple trees, because every leaf in every tree by default is unique.


One approach would be such a set of hyperparameters that would ensure fixed tree structure for all trees in the ensemble, another: re-using the same pre-trained tree-based model object across all cross-validation folds or members of an ensemble, and yet another - a modified predict function that would keep a tree structure intact (ideally also with unchanged split points).

  • $\begingroup$ This will provide no benefit. The wikipedia article on ensemble learning theory says "Empirically, ensembles tend to yield better results when there is a significant diversity among the models." By specifying the tree structure you'll essentially be creating copies of the same model. $\endgroup$
    – Ryan Volpi
    May 7, 2022 at 3:36
  • $\begingroup$ @RyanVolpi yes, I realize it defeats the purpose of ensembling (by forgoing variance reduction), but there are other significant benefits from those tree-based models: 1) built-in cross-validation and 2) computational speed. I think the models should be forced to specialize to just 1 tree to get the above benefits without distorting predictions (in contrast to picking 1 tree from the ensemble, e.g. with lgbm.create_tree_digraph)? $\endgroup$
    – mirekphd
    May 7, 2022 at 7:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.