I have encountered a behavior pattern of lightgbm (same for xgboost), which I do not understand. Below is a code snippet (Python) followed by my explanations:

x = list(range(1000)); y = [0]*500 + [1]*500
import numpy as np
X = np.array(x); X.shape = (len(x), 1)
import lightgbm as lgb
data = lgb.Dataset(X, y)
params = {"objective": "binary", "learning_rate": 1, "num_leaves": 2}
model = lgb.train(params, data, num_boost_round=1)
   threshold     value
0      500.5  0.000000
1        NaN -1.992016
2        NaN  2.000000
Out[2]: {0.12004374604851242, 0.8807970779778823}
set(np.array(y)[model.predict(X) > 0.5])
Out[3]: {1}

This is a simple case of a single decision tree with two leaves, on a single variable which perfectly separates y to 0 and 1. I use binary log loss (the same effect does not happen with l2 loss). What I do not understand is why the values in the leaves are not perfectly 0 and 1, rather they are ~0.12 and ~0.88.

The output Out[1] shows the split threshold and the values in the leaves, after logistic transform. The output Out[2] shows that the tree's predictions are ~0.12 and ~0.88. The output Out[3] shows that the observations for which the tree predicted ~0.88, are all labeled y=1, and so a prediction equal to 1 for this leaf would have been a better fit.


I think your understanding is maturing and you now see the direct reason as to why the learning_rate parameter is also referred as shrinkage_rate. A smaller learning step size stops us from over-fitting by making our boosting process less aggressive. This "less aggression" is implemented by re-weighting our boosting weight values accordingly, as this is usually making the weights' magnitude smaller it is also referred as shrinkage. Let's see how the booster we just defined looks like:

graph_to_save = lgb.create_tree_digraph(model, tree_index=0, orientation='vertical')
graph_to_save.format = 'png'            

enter image description here

And as you correctly note the tree predictions through the inverse-logit transformation are ~0.12 and ~0.88. i.e.

1/(1+np.exp(-(np.array([-1.992, 2])))) 
# array([0.12004543, 0.88079708])

Now if we use a higher learning rate, for example 10, our resulting booster would be like this:

params_10 = {"objective": "binary", "learning_rate": 10, "num_leaves": 2}
model_10 = lgb.train(params_10, data, num_boost_round=1)  
graph_to_save = lgb.create_tree_digraph(model_10, tree_index=0, orientation='vertical')
graph_to_save.format = 'png'            

enter image description here

which obviously we will give us predictions as:

1/(1+np.exp(-(np.array([-19.92, 20.0])))) 
# array([2.23282106e-09, 9.99999998e-01])

which are ~0.0 and ~1.0 as expected. Therefore, going back to your original question as the why "the values in the leaves are not perfectly 0 and 1" the answer is simply that due to shrinkage we stopped our booster from assigned too much weight on the out of a single base learner. Higher learning rates would have resulted to values values much closer to 0 and 1 (when back-transformed).

  • $\begingroup$ Thanks! but shouldn't learning_rate=1 be equivalent to no shrinkage? $\endgroup$
    – erezmb
    May 11 at 8:01
  • $\begingroup$ Unfortunately, it is not but it is close to it! Going back to the definition of shrinkage as learning rate, GBMs take steps across function space. So the concept of zero shrinkage does not really exist as it is directly related to the step size performed. Think of it in the context of a GLM doing IRLS we might have no explicit shrinkage but we still need to make a step to get our new beta coefficients. $\endgroup$
    – usεr11852
    May 11 at 10:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.