0
$\begingroup$

Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in [0,1]$. Assume one use gbdt to learn this linear model. And if the test data consists of ($X,Y$) such that $X \in [4,5]$. Can the learned gbdt model correctly predict data in test set? The reason is that any node in any tree in gbdt has the form: $x_i > a $ and $x_i <=a$, $a$ is the splitting point of variable $x_i$, and the $X$ in test data has disjoint range with the one in train set, so the gbdt model will not be able to distinguish the $X\in [0,1]$ in train set and $X \in [4,5]$ in test set, more specifically, the $X\in [4,5]$ will lie to the right side of the splitting point and then will predict the same value as the $X$ in the upper part of $[0,1]$, for example, in $[2/3,1]$. As the true model is linear, which means it wil not predict the same value for $X$ in train and test set, resulting that the gbdt will not correctly learn such a model. Is the above reasoning correct?

$\endgroup$
  • $\begingroup$ GBDT is a non-standard abbreviation. Please use good practice and define your abbreviations when you first use them. $\endgroup$ – Matthew Drury Aug 26 '18 at 5:30
4
$\begingroup$

You're right.

If your training set contains only points $X \in [0, 1]$, and the test only $X \in [4, 5]$, then ay tree based model will not be able to generalize even a simple linear relationship like $y \approx 2x$ outside of the domain covered by the training set. Instead, the learned model will predict a constant for all $X \in (1, \infty)$, since this range is greater than the largest possible split point for any tree.

This is not really an issue with the trees though. It's an often unstated, but important, assumption of any (*) machine learning model that the training and testing sets are samples from the same population. This means that joint distribution of $X, Y$ should be the same for both the training and testing data sets. The validity of most methods, for example cross validation, rests on this assumption.

(*) Not really any, but most.

$\endgroup$
  • $\begingroup$ Thanks!One point I can think is that we can use transfer learning if train and test data have differernt distributions, but the train and test sets differ only in scale, I am not be able to find any transfer learning method that can deal with this case. Do you have any suggestion?Thanks $\endgroup$ – Hao Yu Aug 26 '18 at 5:51
  • $\begingroup$ The good practice is to use the stacked ensemble. First, a linear model makes predictions - it may be not very accurate, but it is good at extrapolation. Then, the GBDT predict residuals of the linear model, which can increse accuracy dramatically without losing sensitivity to trends. $\endgroup$ – David Dale Aug 26 '18 at 6:31
  • 2
    $\begingroup$ (+1) This is interesting - could you tell me what machine learning methods do not require training/testing sets to be from the same population? $\endgroup$ – mkt Aug 27 '18 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.