Why do my XGboosted trees all look the same?

Question

I am running an XGBRegressor that is supposed to predict a certain reward associated to different actions, which are one-hot encoded. For testing purposes I am using a small depth=2 and only 10 trees:

XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1, colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=2, min_child_weight=1, missing=None, n_estimators=10, n_jobs=1, nthread=None, objective='reg:linear', random_state=0, reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None, silent=True, subsample=1)

The R-SQR is Ok. However, after visualizing the individual trees, it seems that almost all the trees are using exactly the same features (and condition and thresholds being the same), but differ slightly in the resulting leaf values.

What's the point of that? In that case a linear addition of the trees would be sufficient and we'd only have one. No need to have exactly the same trees with different leaf values, right? It's also not the case that the first tree has the largest values, and the remaining trees having insignificant values - they all have similar looking values with the bottom right one being the largest (see image).

So my questions on that are:

1. Why is it doing multiple trees with exactly the same features and equality conditions and slightly different values
2. Is there a way to say: "Stop adding trees if they don't add any more significant value"?

Answer 1

When a gradient boosting machine fits a tree $f(x)$ to the target variable $y$, it calculates the error (used in the next iteration to fit the next tree) as:

$$e = y - \epsilon f(x)$$

where in this case $\epsilon = 0.1$, the learning rate. Now imagine that there is a fairly simple tree $g(x)$ that actually fits the data reasonably well (for a simple tree, that is.) At our first iteration, our model finds $g(x)$ and calculates the errors $e$ for the next step. At that step, $e$ will likely be fit quite well by the same tree, because we didn't subtract all of $g(x)$ from $y$, we only subtracted $10\%$ of it. Writing rather loosely, $90\%$ of $g(x)$ is still in the error term $e$. Because $e \neq y$, the leaf values will be a little different, but the tree structure will be the same, so at the next iteration we'll fit $g(x)$ again.

We may have to do this quite a few times before we have done enough fitting of $g(x)$ to $y$ for other tree structures to become obvious enough for the GBM to find in preference to more fitting of $g$.

The fundamental cause of this is that we are taking a step towards $g(x)$ of fixed length - $\epsilon$ - not performing a line search to find the optimum step length towards $g(x)$ at each iteration then taking one step of that length. This is a feature! It helps prevent overfitting. If you fit "optimum" trees at every step of the GBM, you very rapidly overfit, even with small trees. (Note that they aren't really optimum trees, as they are found through a greedy search heuristic.) Random Forests do fit "optimum" trees at every step, and deep ones too, but deal with the overfitting issue by averaging across trees. GBM takes a different approach, one which may appear to waste iterations in cases such as yours, but in general works quite well on a broad range of problems.