5
$\begingroup$

I have a relatively good understanding of how RandomForest mechanically works. However, here's what I want to understand: can RF model a multiplicative relationship?

For example, if I have features A and B predicting target variable C. Let's say the true relationship is A x B=C . Would RF be able to model this? Or would I have to have (A x B) as an input feature?

Thank you!

$\endgroup$
3
  • $\begingroup$ Random forests can't even model additive relationships in the sense of reproducing them accurately within nonempty intervals. In what sense, then, do you mean "model" in this context? $\endgroup$
    – whuber
    Commented May 2, 2022 at 19:39
  • $\begingroup$ @whuber Fair point. I guess I just mean, from a practical/reasonable perspective, can I expect an RandomForest to pick up on that kind of relationship without me explicitly including the AxB feature as a column in the dataset. $\endgroup$ Commented May 2, 2022 at 20:29
  • 1
    $\begingroup$ In that respect, you have gotten a good answer! $\endgroup$
    – whuber
    Commented May 2, 2022 at 20:31

1 Answer 1

8
$\begingroup$

A tree can form a lot of splits of the form $A>a_i$ or $B>b_i$. Enough of such splits in a single tree can give you a step function that can then approximate just about any function of $A$ and $B$. In RF you usually limit how deep each tree gets, but instead average over lots of trees, which in combination again can give you a step function in $A$ and $B$ that can then approximate just about any function of $A$ and $B$.

However, if your trees are, say, 2 deep, that just gives you a single step in your step function per tree, so it takes a lot of trees, before you approximate a linear relationship such as $A\times B$ really well. If you give the RF the feature $A\times B$ instead, it needs half the depth for the same approximation. I.e. the functional relationship becomes easier for the model to fit.

Nevertheless, if you don't have all that much training data, that will limit how smooth the output of the RF can be no matter whether it gets $A$ and $B$, or additionally $A \times B$ as features.

$\endgroup$
1
  • 1
    $\begingroup$ This totally makes sense, thank you. I'm working with ~15,000 data points, so I assume that is a reasonably large amount so I can expect a good degree of smoothness on this relationship. $\endgroup$ Commented May 2, 2022 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.