Is there a random forest variant which handles relationships between variables more elegantly?

Question

Let's assume that I want to train a random forest classifier which predicts 1 if variables a/b = constant and 0 otherwise. If I have enough data, it should be possible to build decision trees which do just that.

However, I wonder if there is a random forest variant which automatically tries to combine two variables in different ways to see if there are more optimal split points.

Surely, if there are only few variables, I could just generate such feature combinations beforehand and use a standard random forest implementation. However, with lots of features, this quickly becomes a problem. The random forest implementation, on the other hand, could just pick random features and combine them on the fly.

Is anybody aware of a random forest implementation which does just that or research papers about similar ideas?

Answer 1

Random forests (RF) are essentially implementing an ensemble learning approach where the learner perform recursive partitioning of the training sample's feature space. Interactions between variables are captured as a variable $x_b$ being used to partition a subspace already partition by a variable $x_a$. One of RF's strength is exactly this, it automatically partition the training sample such that if such an interaction between $x_a$ and $x_b$ is relevant, it will be automatically picked up. We do not have to code this interaction.

What you describe seems to be resemble something like symbolic regression where the "optimal" set of mathematical operations would be picked as the final model but this is not part of any RF-like approach. Critically while symbolic regression (and genetic programming in general) holds great promise it has not delivered yet. There are some approaches like the GA2M (e.g. see "Accurate Intelligible Models With Pairwise Interactions" by Lou et al. (KDD 2013)) that try to account for two-way interactions more explicitly, but this again is not a RF approach but rather a GAM-based approach.

Finally, I would also note that RFs are actually used directly to detect variable interactions on their own right (e.g. "Iterative random forests to discover predictive and stable high-order interactions" by Basu et al (PNAS 2018) or "Do little interactions get lost in dark random forests?" by Wright et al. (BMC Bioinformatics 2016)). Therefore, I would say that there are no (obvious?) random forest variants that account for interactions explicitly; "standard" RF methodology works satisfactory in handling interactions and creating relevant "feature combinations" as it is.

(Side-note: A relation like the one initially mentioned $C>\frac{x_a}{x_b}$ can could be easily visualised through a two-variable partial dependency plot; it would materialise as a strong diagonal trend. To start with, methodologies like the one mentioned by Basu et al. would allow detecting the relevant $x_a$, $x_b$ interaction.)