Is Gradient Boosting Regression Tree able to learn linear models

Question

Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in ［0,1］$. Assume one use gbdt to learn this linear model. And if the test data consists of ($X,Y$) such that $X \in ［4,5］$. Can the learned gbdt model correctly predict data in test set? The reason is that any node in any tree in gbdt has the form: $x_i > a $ and $x_i <=a$, $a$ is the splitting point of variable $x_i$, and the $X$ in test data has disjoint range with the one in train set, so the gbdt model will not be able to distinguish the $X\in ［0,1］$ in train set and $X \in ［4,5］$ in test set, more specifically, the $X\in ［4,5］$ will lie to the right side of the splitting point and then will predict the same value as the $X$ in the upper part of $［0,1］$, for example, in $［2/3,1］$. As the true model is linear, which means it wil not predict the same value for $X$ in train and test set, resulting that the gbdt will not correctly learn such a model. Is the above reasoning correct?

Answer 1

You're right.

If your training set contains only points $X \in [0, 1]$, and the test only $X \in [4, 5]$, then ay tree based model will not be able to generalize even a simple linear relationship like $y \approx 2x$ outside of the domain covered by the training set. Instead, the learned model will predict a constant for all $X \in (1, \infty)$, since this range is greater than the largest possible split point for any tree.

This is not really an issue with the trees though. It's an often unstated, but important, assumption of any (*) machine learning model that the training and testing sets are samples from the same population. This means that joint distribution of $X, Y$ should be the same for both the training and testing data sets. The validity of most methods, for example cross validation, rests on this assumption.

(*) Not really any, but most.