I can use random forests to determine which features are important when doing a prediction problem; for example.

< height, weight, IQ measure> -> Is considered obese?

Applying random forests could tell me that weight contributes 0.75 percent to obesity.

But what if weight was dependent on height? For example weight = height^2 + IQ Measure^3/height or something?

How would this affect the generated feature importances? Would this 'hidden' relationship have any consequences?

I just thought about this, and was wondering whether this could be an issue. As in, my feature importances would be inaccurate.


It is quite possible that it is an issue e.g. http://www.biomedcentral.com/1471-2105/9/307 the severity of which you can examine using recursive feature elimination http://arxiv.org/abs/1310.5726 but it is less of an issue than in multiple regression as was discussed here earlier Do correlated and/or derived fields require special consideration when using Random Forest?

  • $\begingroup$ From my experience, random forests seems to give me more realistic feature importances. - But in theory should RFE do better? Would you always recommend RFE over RF? $\endgroup$
    – user46925
    Nov 30 '14 at 18:14
  • $\begingroup$ The suggested rfe reference is just a way to explore how much it matters in this particular case, there are other ways. In my experience, RF is very robust to this problem (so I certainly don't recommend supplanting it), but as the first reference shows it is not always immune. And another consideration is why correlated predictors were included: whether they are necessary and meaningful in the context of your model or whether condensing or more carefully chosing them would be more justified. $\endgroup$
    – katya
    Nov 30 '14 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy