I'm wondering if tree based methods are capable of making predictions that are larger in magnitude than the largest training observations? Given my understanding of decision trees and partitioning feature space, I would guess it's not possible for predictions from GBMs or Random Forests to exceed the largest values found in training data.

Assuming the above is true, how do modelers deal with these tree-based limitations. If tree based algorithm predictions are bounded by what's observed in training data, I would imagine this would result in prediction problems for applications where unprecedented values are feasible. Other than employ a different algorithm, is there a different way to handle this boundedness of predictions?

  • $\begingroup$ @Firebug- It answers part 1 of the question. I guess I'm still wondering if modelers ever try to use a different weak learner given the bounded predictions from based methods. $\endgroup$
    – Chris
    Nov 18, 2020 at 20:49
  • $\begingroup$ Your question appears to be answered by the duplicate. If you have a question that's not answered there, then please edit to clarify. $\endgroup$
    – Sycorax
    Nov 19, 2020 at 4:48
  • $\begingroup$ For part 1 with gradient boosting, see stats.stackexchange.com/q/304962/232706 $\endgroup$ Nov 23, 2020 at 15:32


Browse other questions tagged or ask your own question.