I am trying to get a low level understanding of how different machine learning algorithms work and how effective they are.

I understand the basics of random forests: How to build a decision tree by from a subset of features and training set by maximize the information value of each node split. And use many very imperfect trees to build a much more robust model of the data.

But to what extend would it benefit to "help" to use my knowledge of the data to recode it before feeding it to the algorithm?

Example: Say I want to predict how fast a person would be able to run five kilometers considering their sex, weight, age, daily exercise etc. I have prior knowledge that age correlate with the output feature in a way that twenty year olds are the fastest on average and the further away from that people get slower. I could create a new feature AgeDistance" on basis of the age variable that captures that correlation: highest value at age twenty and lowest at each end. I would probably be able to find a good fitting function to describe the age/speed relationship and I would let the age feature stay to catch specific interactions between "real age" and the other features important for the output feature.

But would it do me any good? at the very basic level, when age is a dominant feature among those selected for the tree, you would get same information gain by splitting the training set by some distance to twenty year age than, say, split at fifteen and then later at twenty five but in one node, not two, and retain more examples in the next node for (marginally) better generalisation for the next split. And if a specific age+exercise interaction was superior to that of AgeDistance the latter wouldn´t be selected for the split.

So the question is: Is there any benefit of trying to "prechew" the features for random forests as you would when doing more old school statistical modelling based on your prior knowledge of the data?

up vote 1 down vote accepted

Any monotonic transformation will not change the output of a tree-based algorithm such as random forest. That's because trees choose their splits based on how a feature orders observations, not on is absolute value.

If you apply a non-monotonic transformation (like inverse U-shaped one that you described), output of a tree can change. But if the tree is large enough (and Random Forest can be safely built even with very large trees), it is fully capable of capturing such a U-shaped dependency by itself. So your intervention will probably not help much.

The real change you can do is to engineer some features based on more than one input (e.g. their ratios or linear combinations). Trees look on one feature at a time, so if you combine multiple features into one wisely, it can make the difference.

  • "And Random Forest can be safely built even with very large trees". I didn´t know that. It makes sense that my transformation would have little impact. Thanks. – user2523167 Nov 4 '17 at 10:39
  • One question though. Without "AgeDifference" the fastest ages would end up in one node while each "age leg" would end up separated. There might be some subsequent node split where a feature would be superior if the "age legs" was evaluated together but not if they were separate. But then again some other important feature might only work on one leg and that would suppressed if the legs were evaluated together. I guess I ahev to try it out in practise :) – user2523167 Nov 4 '17 at 10:49

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.