I am running an XGBRegressor that is supposed to predict a certain reward associated to different actions, which are one-hot encoded. For testing purposes I am using a small depth=2 and only 10 trees:
XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1, colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=2, min_child_weight=1, missing=None, n_estimators=10, n_jobs=1, nthread=None, objective='reg:linear', random_state=0, reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None, silent=True, subsample=1)
The R-SQR is Ok. However, after visualizing the individual trees, it seems that almost all the trees are using exactly the same features (and condition and thresholds being the same), but differ slightly in the resulting leaf values.
What's the point of that? In that case a linear addition of the trees would be sufficient and we'd only have one. No need to have exactly the same trees with different leaf values, right? It's also not the case that the first tree has the largest values, and the remaining trees having insignificant values - they all have similar looking values with the bottom right one being the largest (see image).
So my questions on that are:
- Why is it doing multiple trees with exactly the same features and equality conditions and slightly different values
- Is there a way to say: "Stop adding trees if they don't add any more significant value"?