5
$\begingroup$

I have a relatively good understanding of how RandomForest mechanically works. However, here's what I want to understand: can RF model a multiplicative relationship?

For example, if I have features A and B predicting target variable C. Let's say the true relationship is A x B=C . Would RF be able to model this? Or would I have to have (A x B) as an input feature?

Thank you!

Improve this question
$\endgroup$
3
  • $\begingroup$ Random forests can't even model additive relationships in the sense of reproducing them accurately within nonempty intervals. In what sense, then, do you mean "model" in this context? $\endgroup$
    – whuber ♦
    May 2, 2022 at 19:39
  • $\begingroup$ @whuber Fair point. I guess I just mean, from a practical/reasonable perspective, can I expect an RandomForest to pick up on that kind of relationship without me explicitly including the AxB feature as a column in the dataset. $\endgroup$
    – Vladimir Belik
    May 2, 2022 at 20:29
  • 1
    $\begingroup$ In that respect, you have gotten a good answer! $\endgroup$
    – whuber ♦
    May 2, 2022 at 20:31

1 Answer 1

Reset to default
8
$\begingroup$

A tree can form a lot of splits of the form $A>a_i$ or $B>b_i$. Enough of such splits in a single tree can give you a step function that can then approximate just about any function of $A$ and $B$. In RF you usually limit how deep each tree gets, but instead average over lots of trees, which in combination again can give you a step function in $A$ and $B$ that can then approximate just about any function of $A$ and $B$.

However, if your trees are, say, 2 deep, that just gives you a single step in your step function per tree, so it takes a lot of trees, before you approximate a linear relationship such as $A\times B$ really well. If you give the RF the feature $A\times B$ instead, it needs half the depth for the same approximation. I.e. the functional relationship becomes easier for the model to fit.

Nevertheless, if you don't have all that much training data, that will limit how smooth the output of the RF can be no matter whether it gets $A$ and $B$, or additionally $A \times B$ as features.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This totally makes sense, thank you. I'm working with ~15,000 data points, so I assume that is a reasonably large amount so I can expect a good degree of smoothness on this relationship. $\endgroup$
    – Vladimir Belik
    May 2, 2022 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.