The usual implementation of decision trees, and estimators based on ensembles of decision trees, use decision trees that threshold on a single predictor at each split. So they divide the feature space into perpendicular hyperlanes.
Is there any benefit to generalizing this a tiny bit. So at each split, the decision tree uses 2 features, and thresholds based on whether $ax_1 + bx_2 \ge T$. In other words, the feature space is now getting divided into hyperplanes that are not necessarily perpendicular.
I think the main disadvantages would be:
- increased computation, and
- overfitting since now using more degrees of freedom
I guess to help both of these issues, randomization could be used analogous to extra trees. The pair of features could be randomly selected, instead of exhaustively trying every pair, and the threshold T and parameters a and b could be chosen randomly as well (perhaps from a set of few discrete values such that the hyperplanes could have a few orientations).
Are there some cases where you think these non-perpendicular decision trees would be an improvement over the regular implementation of various decision tree algorithms?