Let's assume that I want to train a random forest classifier which predicts 1 if variables a/b = constant and 0 otherwise. If I have enough data, it should be possible to build decision trees which do just that.

However, I wonder if there is a random forest variant which automatically tries to combine two variables in different ways to see if there are more optimal split points.

Surely, if there are only few variables, I could just generate such feature combinations beforehand and use a standard random forest implementation. However, with lots of features, this quickly becomes a problem. The random forest implementation, on the other hand, could just pick random features and combine them on the fly.

Is anybody aware of a random forest implementation which does just that or research papers about similar ideas?


1 Answer 1


Random forests (RF) are essentially implementing an ensemble learning approach where the learner perform recursive partitioning of the training sample's feature space. Interactions between variables are captured as a variable $x_b$ being used to partition a subspace already partition by a variable $x_a$. One of RF's strength is exactly this, it automatically partition the training sample such that if such an interaction between $x_a$ and $x_b$ is relevant, it will be automatically picked up. We do not have to code this interaction.

What you describe seems to be resemble something like symbolic regression where the "optimal" set of mathematical operations would be picked as the final model but this is not part of any RF-like approach. Critically while symbolic regression (and genetic programming in general) holds great promise it has not delivered yet. There are some approaches like the GA2M (e.g. see "Accurate Intelligible Models With Pairwise Interactions" by Lou et al. (KDD 2013)) that try to account for two-way interactions more explicitly, but this again is not a RF approach but rather a GAM-based approach.

Finally, I would also note that RFs are actually used directly to detect variable interactions on their own right (e.g. "Iterative random forests to discover predictive and stable high-order interactions" by Basu et al (PNAS 2018) or "Do little interactions get lost in dark random forests?" by Wright et al. (BMC Bioinformatics 2016)). Therefore, I would say that there are no (obvious?) random forest variants that account for interactions explicitly; "standard" RF methodology works satisfactory in handling interactions and creating relevant "feature combinations" as it is.

(Side-note: A relation like the one initially mentioned $C>\frac{x_a}{x_b}$ can could be easily visualised through a two-variable partial dependency plot; it would materialise as a strong diagonal trend. To start with, methodologies like the one mentioned by Basu et al. would allow detecting the relevant $x_a$, $x_b$ interaction.)

  • $\begingroup$ Thank you for the detailed answer. You gave me a new idea: If it is possible to quantify relations of two variables with random forests, then I could just train a random forest first, take all pairs which seem to interact, create new features for them and train a new random forest. Do you know by chance how I can calculate scores for variable pair interactions? $\endgroup$
    – Joel
    Dec 30, 2018 at 7:47
  • $\begingroup$ You could try that, yes. The Wright et al. 2016 paper mentions some relevant metrics, e.g. Pairwise permutation importance (PPI) and Joint importance by maximal subtrees (JIMS), that are available through R packages ( ranger and randomForestSRC respectively). (I think PPI requires a particular ranger version while JIMS is directly available through the find.interaction function.) $\endgroup$
    – usεr11852
    Dec 30, 2018 at 21:31
  • $\begingroup$ Thank you. I will have to see if I can port this functionality to scikit-learn somehow. $\endgroup$
    – Joel
    Dec 31, 2018 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.