2
$\begingroup$

For various regression tree algorithms (e.g. GBM, Random Forest, Extra Trees), is there any sensible way to get predictions for new data when the independent variables for the new cases are much larger than anything seen in training AND there is a highly positive or negative relationship between the independent variables and the response?

For example, here's what I'd like to do:

x = [1, 2, 3]; y = [2.1, 3.9, 5.8]  # y is just about 2 times x
model = random.forest(y ~ x)        # run the model
predict(model, [98, 99, 100])       # predict y where x is much larger now
# Results in something like : [5.1, 5.1, 5.1]
# when I'd expect closer to : [196, 198, 200]

I understand that the prediction returned is just going to be something close to the mean value over a set of leaf nodes in the trees (so I'm not looking for an explanation on why this is happening), but this implies that regression tree algorithms are incapable of making good predictions for new feature data far outside the domain seen in training.

Is this true, or are there common ways to account for this and get more reliable out-of-sample predictions for these types of models?

$\endgroup$
4
  • 2
    $\begingroup$ I don't think that you can expect to infer this relationship only from data without further assumptions if your new data is so far outside the original domain. One possible assumption in this case would obviously be a linear relationship (that you expected) but there are many other conceivable curves as well. There are various approaches of combining boosting or trees with linear regression models. Which one of these is reasonable (or even reliable) for your kind of data is hard to predict, though... $\endgroup$ Commented Jul 13, 2015 at 18:57
  • $\begingroup$ Thanks Achim, it does make sense that you could apply boosting to some polynomial models too, but that seems to negate the benefit of not having to worry about structure and nonlinearities that comes with tree models. Do you know of any examples of papers demonstrating hybrid approaches for the two? $\endgroup$
    – Eric Czech
    Commented Jul 14, 2015 at 19:48
  • 1
    $\begingroup$ As I said before: I don't see how you can you accomplish what you want without strong enough assumptions. As for model-based boosting see e.g. the work of Bühlmann & Hothorn (R package mboost). Model-based trees are available through Loh's GUIDE or our MOB algorithm among others. $\endgroup$ Commented Jul 14, 2015 at 21:40
  • 1
    $\begingroup$ As a general observation, it's hard to decide what the correct behavior for a model should be when one is extrapolating beyond the extrema of the training data. $\endgroup$
    – Sycorax
    Commented Aug 28, 2018 at 16:11

0