Skip to main content
1 of 2
Brian Bien
  • 602
  • 4
  • 21

Can decision trees look multiple levels deep when selecting features to maximize information gain?

Suppose we have two features f1 and f2 that, when examined individually, yield very low or zero information gain relative to competing features. But further suppose that if we were to first split on f1, then f2 would yield high information gain. How can we ensure that the decision tree discovers this split that requires looking two levels deep to realize any information gain?

My concern is that I would not expect the tree to split on f1 or f2 given many competing features with higher individual information gain, and therefore it would fail to discover the optimal f1-f2 combined split (without overfitting by setting its depth and other parameters suboptimally).

When building a regression model based on decision trees (RandomForestRegressor or GradientBoostingRegressor, for example), does one need to explicitly create derived features out of f1 and f2 to ensure that this information is captured?

Brian Bien
  • 602
  • 4
  • 21