0
$\begingroup$

I can use random forests to determine which features are important when doing a prediction problem; for example.

< height, weight, IQ measure> -> Is considered obese?

Applying random forests could tell me that weight contributes 0.75 percent to obesity.

But what if weight was dependent on height? For example weight = height^2 + IQ Measure^3/height or something?

How would this affect the generated feature importances? Would this 'hidden' relationship have any consequences?

I just thought about this, and was wondering whether this could be an issue. As in, my feature importances would be inaccurate.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It is quite possible that it is an issue e.g. http://www.biomedcentral.com/1471-2105/9/307 the severity of which you can examine using recursive feature elimination http://arxiv.org/abs/1310.5726 but it is less of an issue than in multiple regression as was discussed here earlier Do correlated and/or derived fields require special consideration when using Random Forest?

Improve this answer
$\endgroup$
2
  • $\begingroup$ From my experience, random forests seems to give me more realistic feature importances. - But in theory should RFE do better? Would you always recommend RFE over RF? $\endgroup$
    – user46925
    Commented Nov 30, 2014 at 18:14
  • $\begingroup$ The suggested rfe reference is just a way to explore how much it matters in this particular case, there are other ways. In my experience, RF is very robust to this problem (so I certainly don't recommend supplanting it), but as the first reference shows it is not always immune. And another consideration is why correlated predictors were included: whether they are necessary and meaningful in the context of your model or whether condensing or more carefully chosing them would be more justified. $\endgroup$
    – katya
    Commented Nov 30, 2014 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.