The accumulative tree structure in a tree based gradient boosting

Question

I'm playing with gradient boosting methods and with its python packages out there. I tried lightgbm, started with a high-dimensional input to predict a task. A left the training settings to default (which means n_estimator=100 and n_leaves=31) and fitting was done without anything going wrong. I assumed that at each step, it makes a 31-leaves tree to fit inputs to the (pseudo-)residuals, find the proper $\gamma_{jm}$ for each partition of the $m$th tree and then up the results with some shrinkage rate $\nu$ in which I have used the wikipedia's notation.

So to me, after fitting, I should have 100 trees, each partitioning the input into 31 sections and estimation a score for that. Yet the splitting point over each tree can be different. Therefore, I expected the resultant estimator to have partitioned my input into n_estimator * n_leaves + 1 regions and have attributed score over each single one of them.

I tried to visualize the tree and found out that the resultant tree only has 31 leaves. How can it be? What am I missing?

Answer 1

Your description sounds like the algorithm is working correctly. Each tree can have up to 31 leaves, depending on other parameters like max_depth and min_data_in_leaf. Visualizing the last tree, or any tree, will show up to 31 leaves.

Perhaps this is where the question lies. During scoring, the score for each record is NOT the score from the final tree. GBMs are additive. The score is accumulated from each tree in the GBM. The GBM is not trained to make the last tree the best score. In this case, there are NOT 31 scores that may be given. The scores returned are the accumulation of the scores in each leaf in each tree for each record.

I do not know lightgbm, but if it has a predict by tree, run that for 1 record and and watch the score over each tree.

When scoring, there is nothing special about the last tree or the first tree or any tree. The trees are trained on the residuals in order with any regularization. But then the trees are just trees whose score is accumulated to yield the final score.