Effects of recoding non linear variables for random forest

Question

I am trying to get a low level understanding of how different machine learning algorithms work and how effective they are.

I understand the basics of random forests: How to build a decision tree by from a subset of features and training set by maximize the information value of each node split. And use many very imperfect trees to build a much more robust model of the data.

But to what extend would it benefit to "help" to use my knowledge of the data to recode it before feeding it to the algorithm?

Example: Say I want to predict how fast a person would be able to run five kilometers considering their sex, weight, age, daily exercise etc. I have prior knowledge that age correlate with the output feature in a way that twenty year olds are the fastest on average and the further away from that people get slower. I could create a new feature AgeDistance" on basis of the age variable that captures that correlation: highest value at age twenty and lowest at each end. I would probably be able to find a good fitting function to describe the age/speed relationship and I would let the age feature stay to catch specific interactions between "real age" and the other features important for the output feature.

But would it do me any good? at the very basic level, when age is a dominant feature among those selected for the tree, you would get same information gain by splitting the training set by some distance to twenty year age than, say, split at fifteen and then later at twenty five but in one node, not two, and retain more examples in the next node for (marginally) better generalisation for the next split. And if a specific age+exercise interaction was superior to that of AgeDistance the latter wouldn´t be selected for the split.

So the question is: Is there any benefit of trying to "prechew" the features for random forests as you would when doing more old school statistical modelling based on your prior knowledge of the data?

Answer 1

Any monotonic transformation will not change the output of a tree-based algorithm such as random forest. That's because trees choose their splits based on how a feature orders observations, not on is absolute value.

If you apply a non-monotonic transformation (like inverse U-shaped one that you described), output of a tree can change. But if the tree is large enough (and Random Forest can be safely built even with very large trees), it is fully capable of capturing such a U-shaped dependency by itself. So your intervention will probably not help much.

The real change you can do is to engineer some features based on more than one input (e.g. their ratios or linear combinations). Trees look on one feature at a time, so if you combine multiple features into one wisely, it can make the difference.